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'Greater  Use  Of  Exemplary  Education  f 
Programs  Could  Improve  Education  j 
For  Disadvantaged  Children.  . _ 


Although  low-achieving  students  in  many  of 
the  Nation's  federally  funded  compensatory 
elementary  and  secondary  education  projects 
have  improved  their  performance,  large  per¬ 
centages  of  students  in  some  projects  continue 
to  fall  further  behind  their  peers. 

The  National  Diffusion  Network  is  the  pri¬ 
mary  system  the  Department  of  Education 
uses  in  meeting  the  congressional  mandate  to 
disseminate  information  about  successful  edu¬ 
cation  practices  in  State  and  local  education 
agencies  to  improve  the  quality  and  effective¬ 
ness  of  Federal  programs.  Although  some 
school  districts  have  adopted  the  Network's 
exemplary  projects,  the  number  of  adoptions 
is  too  small  to  greatly  affect  the  overall  qual¬ 
ity  of  the  Nation's  compensatory  education 
projects.  A  greater  effort  should  be  made  to 
identify  projects  needing  improvement,  and 
better  data  on  the  effectiveness  of  the  Net¬ 
work's  exemplary  projects  should  be  made 
available  to  school  officials.), 
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To  the  President  of  the  Senate  and  the 
Speaker  of  the  House  of  Representatives 

This  report  discusses  the  potential  for  improving  education 
for  disadvantaged  children  through  greater  use  of  exemplary 
education  programs  available  through  the  Department  of  Educa¬ 
tion'  s  National  Diffusion  Network.  We  examined  the  Network's 
efforts  to  promote  the  adoption  of  exemplary  programs  by  local 
school  districts  to  improve  reading  projects  conducted  under 
title  I  of  the  Elementary  and  Secondary  Education  Act. 

Copies  of  the  report  are  being  sent  to  the  Director ,  Office 
of  Management  and  Budget,  and  the  Secretary  of  Education. 
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of  the  United  States 


Accession  For 

WTIS  GRA&I 
DTIC  TAB 
Unannounced 
Justification 


By - 

Distribution/ 
Availability  Codes 


Avail  and/or 

Dist 

ft 

Special 

COMPTROLLER  GENERAL'S  GREATER  USE  OF  EXEMPLARY  EDUCATION 

REPORT  TO  THE  CONGRESS  PROGRAMS  COULD  IMPROVE  EDUCATION 

FOR  DISADVANTAGED  CHILDREN 


DIGEST 

Federal  programs  for  elementary  and  secondary  edu¬ 
cation  are  intended  to  help  disadvantaged  students 
acquire  basic  education  skills,  improve  their  rate 
of  academic  achievement,  and  ultimately  help  them 
become  self-sufficient  and  self-supporting.  By  far 
the  largest  of  these  programs  is  that  authorized 
under  title  I  of  the  Elementary  and  Secondary 
Education  Act.  Title  I  was  envisioned  as  a  potent 
tool  for  dealing  with  poverty  and  its  attendant 
conditions,  such  as  illiteracy,  high  dropout  rates, 
delinquency,  and  crime.  (See  p.  1.) 

To  improve  the  effectiveness  of  these  programs,  the 
Congress  mandated  that  the  Department  of  Education 
(ED)  disseminate  information  about  successful 
education  practices  to  State  and  local  education 
agencies . 

The  National  Diffusion  Network,  established  by  ED 
in  1974,  disseminates  information  on  various  "ex¬ 
emplary"  education  projects  that  have  been  found 
highly  effective.  The  Network  is  ED's  primary 
mechanism  for  disseminating  information  on  exem¬ 
plary  title  I  projects  and  for  helping  local  school 
districts  to  implement  such  projects.  GAO  reviewed 
the  Network' s  performance  in  helping  local  school 
districts  to  improve  their  title  I  projects  by 
adopting  exemplary  projects.  (See  pp.  2,  6,  and 
7.) 

OPPORTUNITIES  FOR  IMPROVING 
TITLE  I  BY  ADOPTING 
EXEMPLARY  PROJECTS 

Although  many  title  I  reading  projects  are  improv¬ 
ing  the  academic  achievements  of  educationally  dis¬ 
advantaged  students,  some  projects  are  not  success¬ 
ful  and  should  be  improved.  GAO's  analysis  of  stu¬ 
dent  progress  in  about  340  schools  in  14  districts 
revealed  that  many  students  continue  to  fall  behind 
their  normal  (average)  achieving  peers.  Title  I 
services  are  discontinued  for  most  students  while 
they  are  still  far  behind  their  peers  in  academic 
achievement.  Furthermore,  many  needy  students  are 
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excluded  from  receiving  title  I  services  because 
of  the  high  cost  of  some  projects  and  limitations 
on  the  amount  of  available  funds.  (See  pp.  8. 

9,  11,  and  12.) 

GAO's  comparisons  of  the  accomplishments  of 
title  I  projects  with  the  reported  capabilities 
of  the  Network*  s  exemplary  title  I  projects 
indicated  that  many  school  districts  could  im¬ 
prove  their  title  I  services  by  adopting  ex¬ 
emplary  projects.  Not  only  were  the  exemplary 
projects  reporting  greater  achievements,  but 
they  sometimes  cost  less  and  could  serve  more 
students.  (See  pp.  9  to  11.) 

GREATER  EFFORTS  NEEDED  TO 
ENCOURAGE  ADOPTION  OF 
EXEMPLARY  TITLE  I  PRACTICES 

Some  school  districts  have  adopted  exemplary 
projects  for  title  I  settings,  but  the  number  of 
adoptions  is  too  small  to  have  any  significant 
impact  on  improving  the  effectiveness  of  title  I 
in  meeting  its  nationwide  goals.  (See  p.  17.) 

Evaluation  criteria  and  State 
monitoring  need  improvement 

Title  I  administration  has  shortcomings  at  the 
Federal,  State,  and  local  levels.  ED  has  not 
prescribed  adequate  criteria  for  State  and  local 
title  I  officials  to  use  in  assessing  their  proj¬ 
ects,  and  Federal  monitoring  has  been  weak  and 
has  failed  to  emphasize  project  quality.  Sim¬ 
ilarly,  State  education  agencies  have  provided 
limited  monitoring  of  local  project  results  and 
have  not  ensured  the  adequacy  of  project  evalua¬ 
tions  and  the  effectiveness  of  title  1  projects. 

Without  adequate  Federal  and  State  emphasis  on 
project  quality,  relatively  few  local  school  dis¬ 
tricts  have  sought  to  improve  title  Z  projects. 
Local  title  I  officials  generally  have  not  used 
project  evaluations  in  project  planning  and  im¬ 
provement  and,  in  some  cases,  have  not  developed 
local  evaluations  that  are  suitable  for  identify¬ 
ing  project  weaknesses.  (See  pp.  17  and  18.) 
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Information  is  needed 
on  accomplishments 
of  exemplary  projects 

Relatively  little  information  is  available  about 
the  capabilities  and  accomplishments  of  the  Net¬ 
work'  s  exemplary  projects.  Therefore,  local 
school  district  officials  are  not  convinced  that 
exemplary  projects  are  better  than  their  own,  and 
few  school  districts  have  sought  to  implement 
these  projects.  (See  p.  28.) 

Increased  followup  of  project 
adoptions  is  needed 

Little  is  known  about  the  achievements  of  school 
districts  which  have  adopted  exemplary  projects 
because  the  Network  has  made  little  attempt  to 
follow  up  on  project  adoptions. 

The  Network*  s  revised  operating  instructions  call 
for  monitoring  and  evaluating  project  adoptions  and 
for  providing  data  on  project  outcomes  at  the  adop¬ 
tion  sites.  The  revised  instructions,  however,  do 
not  specify 

— the  nature  of  monitoring  and  evaluation  to  be 
performed  or 

— the  data  to  be  collected  and  reported  on  project 
outcomes.  (See  p.  30.) 

Limited  data  about  results  also  hampered  Network 
officials  in  assessing  the  Network' s  impact  on 
the  title  I  program.  Because  the  Network  did 
not  have  uniform  reporting  requirements  for  its 
operations,  it  did  not  have  reliable  information 
on  the  number  of  exemplary  project  adoptions 
that  served  title  I  projects.  (See  p.  33.) 

RECOMMENDATIONS 


To  improve  the  procedures  for  identifying  and 
correcting  title  I  project  weaknesses,  the 
Secretary  of  Education  should 

— provide  substantially  increased  guidance  and 
technical  assistance  to  State  and  local  school 
officials  in  developing  criteria  for  assessing 
the  effectiveness  of  title  I  projects; 
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— expand  ED's  monitoring  efforts  for  assessing 
whether  State  education  agencies  are  (1)  meeting 
their  responsibilities  to  evaluate  the  effective¬ 
ness  of  local  title  I  projects  and  (2 )  helping 
local  school  districts  to  improve  their  title  I 
projects ; 

—direct  State  education  agencies  to  (1)  assess 
the  validity  of  procedures  local  school  dis¬ 
tricts  use  to  evaluate  their  title  I  projects, 

(2)  prescribe  corrective  measures  where  pro¬ 
cedures  are  inadequate,  and  (3)  help  local 
school  officials  use  evaluations  for  detecting 
and  correcting  project  weaknesses;  and 

—advise  State  education  agencies  to  encourage 
local  school  officials  to  use  Network  assistance 
in  improving  title  I  projects. 

To  convince  potential  adopters  of  the  merits  of 
exemplary  projects,  the  Secretary  should  direct 
the  Network  to  provide  complete,  current  informa¬ 
tion  about  the  capabilities  and  accomplishments 
of  the  Network' s  exemplary  projects,  including 
(1)  their  impact  on  scholastic  achievement,  stu¬ 
dent  behavior,  and  sustained  growth,  (2)  their 
cost  effectiveness,  and  (3)  the  results  of  past 
adoptions  by  other  schools. 

To  provide  more  accurate  data  on  the  volume  of 
title  I  adoptions  and  ensure  that  adoptions 
are  successful,  the  Secretary  should  make  sure 
that  the  Network' s  plans  to  require  followup  on 
project  adoptions  are  adequate  to  ( 1)  provide 
accurate  data  on  the  number  of  adoptions  and  the 
schools  affected,  (2)  determine  whether  the 
projects  have  been  installed  correctly  and  are 
functioning  properly,  and  (3)  provide  for  addi¬ 
tional  implementation  assistance  where  needed. 

(See  pp.  36  and  37.) 

AGENCY  COMMENTS 


In  a  draft  of  this  report,  GAO  proposed  that  ED 
prescribe  specific  criteria  for  State  and  local 
officials  to  use  in  assessing  the  effectiveness 
of  title  I  projects.  In  its  response  to  the 
draft  (see  app.  Ill),  ED  disagreed  with  GAO's 
proposal.  ED  believes  that  its  October  12,  1979, 
title  I  evaluation  regulations,  which  provide 
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models  and  technical  standards  for  evaluating 
title  I  projects,  are  adequate  criteria  on 
evaluation. 

GAO  believes  that  ED's  obligation  in  fulfilling  its 
requirements  does  not  end  with  the  publication  of 
evaluation  models  and  technical  standards.  ED  should 
place  increased  emphasis  on  providing  technical  as¬ 
sistance  at  the  State  and  local  levels  in  developing 
criteria  for  assessing  the  effectiveness  of  title  I 
projects.  This  would  be  consistent  with  ED's  long¬ 
standing  policy  of  nonintervention  in  the  program¬ 
matic  decisionmaking  process  at  the  State  and  local 
levels . 

According  to  ED,  implementation  of  the  title  I 
evaluation  system  will  permit  local  school  dis¬ 
tricts  to  assess  their  title  I  achievement  gains 
on  the  same  metric  or  scale  and  therefore  permit 
States  to  review  results  across  school  districts. 

ED  said  that  national  aggregation  will  enable  State 
and  local  education  agencies  to  view  the  levels  of 
their  gains  in  light  of  the  ranges  of  gains  re¬ 
ported  nationally.  Additionally,  ED  should  use  the 
national,  regional,  State,  and  district  level  infor¬ 
mation  that  it  collects  to  provide  increased  guid¬ 
ance  to  State  and  local  officials  concerning  cri¬ 
teria  for  assessing  the  effectiveness  of  title  I 
projects;  GAO  has  revised  its  recommendation  ac¬ 
cordingly.  ED  agreed  with  the  thrust  of  GAO's 
other  recommendations.  ED's  comments  are  discussed 
on  pages  37  through  42. 
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CHAPTER  1 


INTRODUCTION 

To  improve  the  quality  and  effectiveness  of  Federal  programs 
to  assist  educationally  disadvantaged  children,  the  Congress  man¬ 
dated  that  the  Department  of  Education  (ED)  identify  successful 
education  programs  and  disseminate  information  about  them  to 
States  and  local  school  systems.  To  meet  this  mandate,  ED  uses 
the  National  Diffusion  Network  as  its  principal  system  to  dissemi¬ 
nate  information  about  exemplary  education  projects  and  to  help 
schools  implement  such  projects. 

'  Recognition  of  the  national  need  to  assure  that  educationally 
disadvantaged  children  are  provided  suitable  educational  opportuni¬ 
ties  has  spawned  a  number  of  Federal  programs  to  assist  education 
in  elementary  and  secondary  schools.  Federal  programs  for  elemen¬ 
tary  and  secondary  education  generally  are  intended  to  help  the 
disadvantaged  to  develop  their  basic  skills,  improve  their  rates 
of  academic  achievement,  and  help  them  become  self-sufficient  and 
self-supporting . 

The  largest  Federal  program  for  elementary  and  secondary 
education  comes  under  title  I  of  the  Elementary  and  Secondary 
Education  Act  of  1965  (ESEA)  (20  U.S.C.  241a).  The  Congress, 
recognizing  the  link  between  inadequate  educational  opportunities 
and  poverty,  authorized  the  title  I  program  to  assist  educationally 
disadvantaged  children.  Title  I  was  envisioned  as  a  potent  tool 
for  dealing  with  poverty  and  its  attendant  conditions,  such  as 
illiteracy,  high  dropout  rates,  delinquency,  unemployment,  and 
crime. 

Under  title  I,  funds  are  provided  to  States  and  local  school 
districts  to  help  them  provide  compensatory  education  programs  for 
educationally  disadvantaged  youths.  These  programs  focus  on  de¬ 
veloping  basic  skills  in  reading,  language  arts,  and  mathematics. 
Since  the  program's  inception  in  1965,  funding  has  totaled  about 
$30  billion.  For  fiscal  year  1981  about  $3.7  billion  has  been 
provided  for  the  program,  which  is  expected  to  reach  about  14,000 
of  the  Nation's  approximately  16,000  school  districts. 

In  its  desire  to  improve  the  quality  of  the  Federal  education 
programs,  the  Congress  recognized  the  need  for  dissemination  of 
effective  educational  practices.  Congressional  mandates  to  dis¬ 
seminate  information  to  improve  education  programs  now  appear  in 
various  laws.  Title  I  of  ESEA  mandates  that  ED  (1)  develop  proce¬ 
dures  for  identifying  successful  educational  projects  and  practices 
and  (2)  disseminate  information  on  such  projects  and  practices  to 
States  and  local  school  districts. 


r 


ED  DISSEMINATION  ACTIVITIES 

The  National  Diffusion  Network--the  largest  of  ED's 
dissemination  mechanisms — is  responsible  for  disseminating  in¬ 
formation  on  a  wide  range  of  education  projects  in  such  diverse 
areas  as  bilingual  and  immigrant  education,  career  and  vocational 
education,  early  childhood  education,  environmental  education, 
special  education  for  the  handicapped,  arts  and  communication 
skills,  mathematics,  reading,  and  language  arts.  The  Network  is 
also  ED's  primary  means  for  disseminating  information  on  exemplary 
title  I  projects  and  practices. 

Since  its  inception  in  1974,  the  Network  funding  has  totaled 
about  §59  million.  As  shown  in  the  graph  on  the  following  page, 
funding  has  been  somewhat  erratic,  averaging  about  $7.4  million 
per  year  but  ranging  from  zero  in  1976  to  $11.5  million  in  1979. 

Of  the  Network  funds,  ED  officials  report  that  about  $1.25  to 
$1.75  million  per  year  is  directed  to  title  I  dissemination  ac¬ 
tivities.  By  comparison,  the  annual  title  I  budget  now  exceeds 
$3  billion. 

Before  fiscal  year  1976,  the  Network  was  funded  under 
title  III,  section  306,  of  ESEA,  as  amended.  This  legislation 
expired  at  the  end  of  fiscal  year  1975,  and  the  Network  was  not 
funded  in  fiscal  year  1976.  In  fiscal  year  1977,  the  Network  was 
funded  under  section  422a  of  the  General  Education  Provisions  Act. 
Beginning  with  fiscal  year  1980,  authority  for  Network  funding  was 
contained  in  sections  303  and  376  of  ESEA. 

The  Network's  overall  goal,  as  prescribed  by  ED,  is  to  im¬ 
prove  education  practices  by  broadcasting  information  about 
successful  projects  and  practices  and  assisting  in  the  widespread 
implementation  of  these  projects  and  practices.  ED  instructs 
Federal  and  State  education  program  officers  to  be  alert  to 
identify  highly  successful  education  projects,  including  title  I 
projects,  and  to  encourage  the  project  developers  to  seek  valida¬ 
tion  by  the  Joint  Dissemination  Review  Panel. 

ED  established  the  Joint  Dissemination  Review  Panel  to  review 
data  on  educational  projects  and  assess  whether  the  projects  had 
produced  substantial  educational  improvement  and  whether  they 
could  produce  similar  improvements  at  other  locations.  Officers 
of  the  various  ED  program  divisions  may  nominate  identified  proj¬ 
ects  for  panel  review.  Projects  approved  by  the  panel  are  termed 
"exemplary" — they  are  the  only  ones  that  may  be  endorsed  and  dis¬ 
seminated  by  the  Network . 

The  Network's  activities  to  disseminate  information  about 
exemplary  projects  and  to  help  schools  implement  the  projects  are 
carried  out  largely  by  education  project  "developer-demonstrators" 
and  by  dissemination  "facilitators." 


2 


Developer-demonstrators 

School  districts  that  have  exemplary  projects  approved  by 
the  Joint  Dissemination  Review  Panel  can  apply  for  developer- 
demonstrator  funds  from  the  Network.  Developer-demonstrators  are 
typically  education  agencies  that  originated  the  projects  and  are 
responsible  for  helping  school  districts  implement  their  projects. 
The  developer-demonstrators  are  expected  to  provide  information 
describing  their  projects,  develop  materials  needed  to  implement 
their  projects,  demonstrate  their  projects  to  interested  school 
officials,  and  conduct  training  sessions  for  school  personnel. 

Not  all  exemplary  projects  are  selected  to  become  developer- 
demonstrators.  In  selecting  developer-demonstrators.  Network 
officials  attempt  to  acquire  a  wide  variety  of  projects  and  educa¬ 
tional  approaches  for  school  systems  to  choose  from.  By  fiscal 
year  1979,  the  Network  had  awarded  contracts  to  21  developer- 
demonstrators  to  market  and  help  install  exemplary  title  I  projects. 

State  facilitators 


The  Network  also  contracts  with  organizations  in  specific 
geographical  areas  throughout  the  Nation  to  facilitate  the  adoption 
process  by  linking  school  districts  with  developer-demonstrators. 
These  facilitators  usually  are  assigned  to  cover  specific  States 
and  are  called  "State  facilitators." 

A  State  facilitator's  primary  responsibility  is  to  widely 
disseminate  information  to  local  school  systems  within  its  State 
or  area.  Facilitators  acquaint  schools  with  the  exemplary  proj¬ 
ects,  help  schools  determine  how  the  exemplary  programs  might 
benefit  the  schools,  and  provide  a  link  between  the  developer- 
demonstrators  and  schools  that  express  interest  in  adopting 
exemplary  projects. 

Network  accomplishments 

ED's  assessments  of  the  Network  have  generally  been  favor¬ 
able,  based  on  observations  such  as  the  following: 

— The  Network  has  successfully  promoted  a  large  number  of 
adoptions;  ED  reported  5,600  "instances  of  adoption" 
since  the  Network  began  in  1974. 

— The  Network  provided  training  in  exemplary  project  opera¬ 
tions  to  about  25,000  teachers  in  4,200  schools  during  the 
1978-79  school  year. 

— The  adopted  programs  are  highly  regarded  by  the  adopting 
school  officials. 
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— The  demand  for  the  Network 1 s  services  exceeded  its  capacity 
to  provide  them. 

— Existing  exemplary  programs  can  be  installed  for  an  average 
cost  of  $4,000  whereas  developing  such  a  program  costs 
$400,000. 

These  assessments  were  centered  on  the  Network's  overall  ac¬ 
tivities  in  promoting  program  adoptions.  The  assessments  did  not 
distinguish  between  different  types  of  education  programs,  such  as 
title  I  projects. 

ADMINISTRATION  OF  TITLE  I  RESPONSIBILITIES 


Besides  requiring  that  exemplary  title  I  projects  be  identi¬ 
fied  and  disseminated  to  States  and  local  school  districts,  ESEA 
requires  ED  to 

— review  each  State's  title  I  grant  applications  and  assure 
that  program  plans  are  adequate  and  complete, 

— monitor  each  State's  title  I  project  activities  for  com¬ 
pliance  with  legislative  requirements,  and 

— provide  standards  and  models  for  project  evaluations  and 
provide  technical  evaluation  assistance  to  States. 

The  Office  of  Dissemination  and  Professional  Development  and 
the  Office  of  Compensatory  Education  have  major  roles  in  carrying 
out  ED's  title  I  responsibilities.  The  Office  of  Dissemination  and 
Professional  Development  operates  the  Network,  which  is  responsible 
for 


— improving  the  quality  of  education  nationwide,  including 
title  I  projects; 

— disseminating  information  on  successful  title  I  education 
practices; 

— promoting  widespread  adoption  of  successful  title  I  educa¬ 
tion  practices; 

— helping  schools  to  implement  exemplary  education  projects; 

— providing  training  and  technical  assistance  to  adopting 
schools ; 

— evaluating  the  effectiveness  of  the  adoption  process; 
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— providing  postadoption  technical  assistance  to  adopting 
schools;  and 


— selecting  dissemination  facilitators  and  developer- 
demonstrators  to  carry  out  the  Network's  activities. 

Although  actively  promoting  the  improvement  of  title  I  proj¬ 
ects  through  the  adoption  of  exemplary  projects,  the  Network  does 
not  have  any  formal  responsibility  for  conducting  the  title  I 
program.  The  Office  of  Compensatory  Education  is  responsible  for 
the  overall  administration  of  the  program,  which  includes 

— providing  guidance  to  State  and  local  school  officials, 

— reviewing  State  title  I  grant  applications, 

— assuring  that  State  title  I  responsibilities  are 
fulfilled, 

— reviewing  and  assessing  title  I  program  data  submitted 
by  States,  and 

— identifying  successful  education  programs  and  practices. 
OBJECTIVES,  SCOPE,  AND  METHODOLOGY 


Because  of  the  concerns  of  educators  and  evaluators  that 
title  I  projects  are  not  achieving  intended  goals  and  the  explicit 
intent  of  the  Congress  that  ED  and  school  officials  disseminate 
successful  practices  to  improve  title  1  projects,  our  review 
focused  on  the  Network's  efforts  to  improve  title  I  reading  proj¬ 
ects.  Toward  this  end,  we  reviewed  (1)  the  Network's  actions  to 
promote  adoptions  of  exemplary  projects  for  title  I,  (2)  the  suit¬ 
ability  of  data  that  school  districts  have  for  assessing  the  need 
for  title  I  project  improvement,  and  (3)  school  officials'  actions 
to  adopt  better  title  1  projects .  Our  review  focused  on  reading 
projects  because  most  title  I  projects  are  reading-language 
oriented. 

Our  review  included  examinations  of  legislation;  Federal 
regulations;  Network  records,  reports,  and  materials;  State  and 
local  education  agency  procedures,  records,  and  project  assess¬ 
ments;  and  federally  funded  evaluations  of  title  1  and  other 
education  programs.  We  interviewed  officials  at 

— ED's  Office  of  Dissemination  and  Professional  Development, 
its  Office  of  Compensatory  Education,  and  the  National 
Institute  of  Education; 

— State  education  agencies  in  five  States; 


— dissemination  facilitators  in  five  States  under  contract 
with  the  Network; 

— seven  exemplary  project  developer-demonstrators;  and 

— twenty  school  districts  that  had  implemented  exemplary 
title  I  projects. 

Also,  we  visited  47  local  school  districts  in  six  States, 
including  the  above  five  States,  to  ascertain  the  Network's  impact 
on  the  districts.  These  school  districts  were  selected  to  provide 
a  mix  of  small,  medium,  and  large  districts  in  rural  and  urban 
locations.  The  school  districts  we  visited  are  not  a  statistically 
valid  representation  of  all  school  districts  nationwide,  nor,  in 
our  opinion,  was  the  sample  necessarily  nationally  representative. 
However,  these  districts  represent  a  wide  variety  of  sizes,  types, 
locations,  and  environments. 

Chapter  2  of  this  report  assesses  the  potential  for  improving 
title  1  projects.  Chapter  3  examines  the  effectiveness  of  mecha¬ 
nisms  to  identify  title  1  projects  needing  improvement  and  of  the 
Network's  activities  to  promote  adoptions  of  exemplary  projects 
for  title  I.  Our  conclusions  and  recommendations  and  ED's  response 
are  in  chapter  4 . 


CHAPTER  2 


GREATER  USE  OF  EXEMPLARY  PROJECTS 


COULD  IMPROVE  TITLE  I  PROGRAMS 


Low-achieving  students  in  many  of  the  Nation's  title  I  projects 
have  improved  their  reading  achievement  rates.  Some  title  1  proj¬ 
ects,  however,  are  not  producing  significant  achievement  in  reading 
and  need  improvement.  Students  in  these  latter  title  I  projects 
are  continuing  to  fall  further  behind  average  achieving  students. 
Often  title  I  services  terminate  before  students  reach  suitable 
achievement  levels.  In  some  cases,  the  high  cost  of  title  I  proj¬ 
ects  contributed  to  the  premature  termination  of  the  services. 

Through  the  Network,  ED  disseminates  information  on  exemplary 
projects  proven  capable  of  producing  significant  educational 
achievements  that  are  worthy  of  implementation  in  other  schools 
and  sometimes  cost  less  than  existing  title  I  projects.  Our  com¬ 
parisons  of  exemplary  projects  with  title  I  projects  in  school 
districts  we  visited  indicate  that  substantial  improvements  might 
be  realized  through  greater  use  of  exemplary  projects.  Implement¬ 
ing  an  exemplary  project  in  an  adopting  school  district,  however, 
is  a  complicated  process  requiring  staff  training  and  assistance. 


NEED  TO  IMPROVE  EFFECTIVENESS 
OF  EXISTING  PROGRAMS 


Studies  of  the  national  impact  of  title  I  programs  have  raised 
questions  as  to  whether  the  projects  have  improved  students 1 
achievement  rates.  A  1975  ED-funded  study  1/  showed  that,  on  the 
average,  title  I  students  tended  not  to  fall  further  behind  non¬ 
title  I  students  during  the  academic  year.  However,  the  study  also 
showed  that  many  schools  failed  in  their  goals  to  help  disadvan¬ 
taged  students  gain  on  their  peers  and  that  many  students  were 
continuing  to  fall  behind.  ED  reported  in  its  1978  annual  evalua¬ 
tion  report  that  title  I  projects  have  demonstrated  modest  success, 
but  that  much  work  remains . 

A  1980  study  funded  by  ED  2/  compared  the  reading  growth  of 
title  I  students  with  that  of  similarly  disadvantaged  students  who 
did  not  have  any  compensatory  reading  assistance.  This  study  con¬ 
cluded  that  title  I  students  (1)  in  grades  1,  2,  and  3  grew  at  a 
faster  rate  than  disadvantaged  students  not  receiving  compensatory 


1/ "A  Descriptive  and  Analytic  Study  of  Compensatory  Reading 
Programs"  (Trismen,  Waller,  and  Wilder),  1975. 

2/"The  Sustaining  Effects  Study:  An  Interim  Report, "  L.  Carter, 
October  1980 . 
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education  and  (2)  in  grades  4,  5,  and  6  grew  at  the  same  rate  as 
the  unserved  disadvantaged  students. 

Using  a  different  comparison  standard,  we  analyzed  the  reading 
achievement  gains  of  title  I  students  in  about  340  schools  involv¬ 
ing  14  school  districts.  (See  app.  II  for  analysis  methodology.)  1/ 
Our  analysis  used  a  more  stringent  standard  in  that  it  compared  the 
pretest  to  posttest  gain  of  title  I  students  with  students  at  the 
same  grade  level  who  scored  at  the  national  pretest  average.  We 
made  this  comparison  to  ascertain  whether  title  I  projects  were  en¬ 
abling  the  disadvantaged  students  to  gain  on  their  normal  (average) 
achieving  peers,  'll  This  analysis  disclosed  that  many  students 
made  significant  gains  while  in  title  I  projects.  For  example,  the 
reading  achievement  gains  of  60  percent  or  more  of  the  students  in 
four  districts  consistently  equaled  or  exceeded  1  year  per  year  of 
instruction.  However,  in  some  other  districts  significant  portions 
of  the  students  continued  to  fall  behind  the  regular  students.  In 
about  half  of  the  years  analyzed,  40  to  67  percent  of  the  students 
were  falling  further  behind  their  normal  achieving  peers.  At  the 
time  their  title  I  assistance  was  terminated,  the  students  in  our 
sample  were  an  average  of  1.5  years  below  grade  level. 

INCREASED  USE  OF  EXEMPLARY  PROJECTS  COULD 
IMPROVE  EFFECTIVENESS  OF  TITLE  I  PROJECTS 


To  determine  whether  exemplary  projects  could  make  title  I 
projects  more  effective,  we  sought  to  examine  whether  exemplary 
projects  produce  greater  achievements  than  other  title  I  projects 
and  whether  adopting  an  exemplary  project  enables  a  title  I  project 
to  improve  its  achievement  levels. 

Although  comparable  achievement  data  were  not  always  avail¬ 
able  for  title  I  exemplary  projects,  our  analysis  of  data  avail¬ 
able  for  six  projects  3/  indicated  that  exemplary  projects  have 


1/ ED  expressed  concern  about  our  reliance  on  the  grade  equivalent 
metric  or  scale  in  this  report.  According  to  a  report  on  con¬ 
troversies  in  evaluating  compensatory  education  prepared  by  the 
American  Institute  for  Research  for  the  National  Institute  of 
Education,  all  available  metrics  or  scales  have  weaknesses. 
However,  the  report  notes  that  none  of  the  available  metrics  or 
scales  has  the  same  clarity  and  simplicity  of  meaning  as  grade 
equivalents.  Our  use  of  grade  equivalents  is  further  discussed 
in  appendix  II. 

2/The  evaluation  models  developed  by  ED  compare  the  performance 
of  title  I  students  with  similarly  disadvantaged  students  that 
received  no  compensatory  assistance. 

3/All  six  exemplary  projects  were  either  developed  for  tj.tle  I 
projects  or  recommended  by  the  Network  for  use  in  title  I. 
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good  potential  for  improving  the  effectiveness  of  compensatory 
education  in  public  schools.  One  title  I  exemplary  project,  for 
example,  reported  that  85  percent  of  its  students  gain  at  a  rate 
of  more  than  1  year  per  year  of  instruction.  For  comparison,  we 
selected  22  sample  groups  of  title  I  students  in  the  14  school 
districts  where  we  analyzed  title  I  achievement.  (See  app.  II  for 
a  description  of  the  sample  group  selection  process.  )  Students  in 

21  of  the  22  sample  groups  were  not  able  to  attain  the  rate  of 
achievement  reported  by  the  exemplary  project  in  any  of  the  years 
analyzed. 

None  of  the  six  exemplary  projects  reported  average  gain 
rates  below  the  normal  achievement  rate  of  1  year ‘ s  gain  per  year 
of  instruction  expected  for  average  students.  In  contrast,  over 
half  of  the  22  sample  groups  were  below  the  normal  achievement 
rate  in  at  least  1  of  the  years  analyzed  and  2  groups  were  below 
this  rate  in  each  year  analyzed.  One  title  I  exemplary  project 
reported  that  the  mean  gain  of  students  served  was  1.7  years  per 
year  of  instruction.  This  rate  was  not  reached  by  15  of  the 

22  sample  groups  in  any  year  analyzed.  No  sample  group  averaged 
this  rate  in  each  year  analyzed. 

Other  exemplary  projects  also  reported  significant  accomplish¬ 
ments.  One  project  reported  average  gains  ranging  from  1.4  to 
1.9  years  per  year  of  instruction  for  various  sample  groups.  A 
project  that  used  student  tutors  to  assist  title  I  students  re¬ 
ported  gains  ranging  between  1.2  and  3.7  years  per  year  of  in¬ 
struction  for  both  the  tutors  and  the  students  being  tutored. 
Another  exemplary  project  reported  that  it  doubled  the  rate  of 
gain  of  the  students  it  served. 

According  to  ED,  the  Joint  Dissemination  Review  Panel  has 
assessed  the  reported  exemplary  project  gains  and  can  attest  to 
their  validity  and  educational  significance.  Information  accumu¬ 
lated  by  one  exemplary  project  showed  that  adopting  districts  had 
also  achieved  success.  This  exemplary  project,  which  regularly 
provided  followup  assistance  to  adopting  schools,  reported  that 
achievement  gains  of  students  in  21  adopting  school  districts 
ranged  from  1.1  years  in  grades  2  and  3  to  1.8  years  in  grades  7 
and  8. 

USING  EXEMPLARY  PROJECTS 
COULD  BE  LESS  COSTLY 

Many  school  districts  have  title  I  projects  that  cost  more 
than  some  relatively  inexpensive  exemplary  projects.  The  compen¬ 
satory  education  projects  in  the  14  districts  where  we  assessed 
achievement  gains  cost  up  to  §778  per  pupil  per  year.  The  average 
per  pupil  cost  was  §450  per  year,  and  only  one  district  had  a  per 
pupil  cost  below  §300  per  year. 


In  contrast,  the  six  exemplary  projects  mentioned  above  were 
much  less  expensive.  The  most  expensive  exemplary  project  cost 
$281  per  pupil  per  year,  and  the  average  cost  of  the  six  projects 
was  $180  per  pupil,  less  than  half  the  average  cost  of  the  projects 
in  the  14  school  districts.  Thus,  these  exemplary  projects  were 
not  only  more  effective  in  raising  student  achievement,  but  also 
less  costly.  Increased  use  of  such  projects  might  enable  school 
districts  to  serve  more  students  over  longer  periods. 

According  to  ED  officials,  millions  of  children  eligible  for 
title  I  services  are  not  being  served  because  of  limitations  in 
program  funding.  Additionally,  each  year  title  I  services  to  a 
million  or  more  children  are  terminated  prematurely  (i.e.,  before 
the  students  reach  normal  achievement  levels  for  their  age  and 
grade  level).  However,  ED  has  not  assessed  the  potential  for  ad¬ 
dressing  this  problem  through  development  and  replication  of  exem¬ 
plary  projects  that  cost  less  than  existing  title  I  projects.  ED 
officials  pointed  out  that  project  costs  can  vary  widely  because 
of  teacher  salary  differentials  and  other  factors,  such  as  the 
degree  of  reliance  on  low-cost  tutoring  approaches  to  compensatory 
education. 

Eligible  children  not  served 


According  to  ED  officials,  precise  information  is  not  avail¬ 
able  on  the  number  of  low-achieving  students  eligible  for  title  I 
services  but  not  receiving  them.  ED  officials  estimate,  however, 
that  only  about  half  of  the  Nation's  low-achieving  eligible  stu¬ 
dents  receive  title  I  services.  A  1978  study  for  ED  1/  reported 
that  most  of  the  Nation's  low-achieving  elementary  students  do  not 
receive  compensatory  education  from  title  I  or  any  other  source. 
Other  studies  have  reported  that  compensatory  education  beyond  the 
elementary  level  is  almost  nonexistent. 

Only  one  of  the  five  State  education  agencies  we  visited  had 
any  information  on  the  number  of  eligible  students  not  served  by 
title  I  projects.  This  State  reported  that  about  219,000,  or 
65  percent,  of  the  State's  eligible  students  were  not  receiving 
title  I  services. 

Services  terminated  prematurely 

Title  I  services  are  discontinued  for  many  students  while 
they  are  still  far  below  their  peers  in  academic  achievement. 

This  generally  occurs  when  the  students  are  advanced  to  a  grade 
level  not  having  title  I  services  or  are  replaced  by  needier 
students . 


1/ "Students'  Economic  and  Educational  Status  and  Selection  for 
Compensatory  Education,"  V.  Breglio,  et  al.,  1978. 
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ED  officials  testified  1/  that  many  eligible  students  are 
excluded  from  title  I  projects  because  sufficient  funds  are  not 
available.  According  to  ED,  most  title  I  services  are  at  the 
elementary  level,  and  a  high  percentage  of  title  I  funds  are 
used  for  the  first  three  grades.  ED  officials  estimate  that  an 
additional  one-third  to  one-half  of  the  elementary  students  need 
compensatory  services  and  that  the  need  at  the  secondary  level  is 
far  greater. 

Educators  we  talked  to  in  the  five  States  we  visited  ex¬ 
pressed  concern  that  the  need  for  compensatory  education  was  not 
being  met  beyond  the  early  grades.  The  Congress,  concerned  about 
the  high  rate  of  illiteracy,  stipulated  2/  that  school  districts 
should  consider  extending  title  I  services  in  intermediate  and 
secondary  levels  to  sustain  the  gains  of  earlier  services.  Recog¬ 
nizing  the  need  for  compensatory  programs  at  the  secondary  level, 
ED  has  begun  to  publicize  exemplary  programs  appropriate  for 
secondary  schools. 

The  results  of  our  analysis  of  student  achievement  gains 
suggest  that  concern  over  the  progress  of  low-achieving  students 
denied  compensatory  assistance  may  be  well  founded.  Nearly 
11,000  students  in  our  sample  groups  had  their  compensatory 
assistance  terminated  or  substantially  interrupted  (by  1  school 
year  or  more)  during  the  period  analyzed.  When  the  assistance 
was  terminated,  the  students'  rate  of  achievement  gain  declined 
by  an  average  of  0.6  years  per  year  of  instruction.  While  re¬ 
ceiving  compensatory  assistance,  most  of  these  students  were 
gaining  on  their  normal  achieving  classmates.  However,  when  the 
assistance  was  terminated,  the  average  rate  of  gain  dropped  well 
below  the  normal  rate.  This  pattern  of  pronounced  decline  in 
rate  of  achievement  occurred  in  11  of  the  14  school  districts 
analyzed. 

This  decline  in  average  achievement  gain  rate  was  confirmed 
by  a  corresponding  drop  in  the  number  and  percentage  of  students 
gaining  at  or  above  the  normal  rate  of  1  year  per  year  of  instruc¬ 
tion.  As  shown  in  the  following  graph,  the  percentage  of  stu¬ 
dents  gaining  at  or  above  the  normal  rate  for  average  students 
while  receiving  compensatory  reading  assistance  declined  from 
77  to  41  percent  when  assistance  was  terminated. 


l./Hearings  before  a  subcommittee  of  the  House  Committee  on 
Appropriations,  March  15,  1978,  p.  86. 

^/Senate  Report  No.  95-856,  May  15,  1978,  Education  Amendments 
of  1978. 
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PERCENT  OF  STUDENTS  ACHIEVING 
PERCENT  AT  OR  ABOVE  NORMAL  RATES 
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Similarly,  the  percentage  of  students  rapidly  falling  behind 
their  peers  (those  gaining  at  half  the  normal  rate  or  less)  in¬ 
creased  from  14  to  38  percent  when  assistance  was  terminated. 

This  increase  is  shown  in  the  following  graph. 


13 


PERCENT 
50  r 


PERCENT  OF  STUDENTS  FAILING  TO  GAIN  MORE 
THAN  1/2  YEAR  PER  YEAR  OF  INSTRUCTION 
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Compensatory  reading  students  in  the  14  districts  we  analyzed 
were  still  far  behind  their  normal  achieving  classmates  when  their 
compensatory  assistance  ended.  The  average  achievement  level  of 
these  students  was  1.5  years  below  grade  level  when  their  title  I 
assistance  ended.  This  deficit  ranged  from  0.5  to  2.4  years  below 
grade  level  in  the  districts  analyzed. 
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In  one  large  school  district,  title  I  assistance  was  resumed 
for  1,195  students  after  they  had  been  out  of  the  program  for  at 
least  1  school  year.  Their  rates  of  achievement  gain  dropped  sig¬ 
nificantly  when  the  title  I  assistance  was  initially  terminated, 
and  very  few  were  able  to  gain  at  the  normal  rate  or  better  while 
out  of  the  program.  However,  when  title  I  assistance  was  resumed, 
the  achievement  rates  of  many  of  these  students  increased  signifi¬ 
cantly.  As  shown  in  the  following  graph,  the  percentage  of  stu¬ 
dents  keeping  up  with  or  gaining  on  their  peers  jumped  from  6  per¬ 
cent  while  out  of  the  program  to  78  percent  when  assistance  resumed 
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Terminating  title  I  services  before  the  students  attain  accept 
able  achievement  levels  appears  to  conflict  with  the  title  I  goal 
of  assisting  educationally  deprived  students  in  acquiring  the  basic 
skills  needed  to  become  self-sufficient  in  a  competitive  society. 
Services  for  most  title  1  students  are  discontinued  in  the  elemen¬ 
tary  grades,  whereas  most  dropouts  occur  between  grades  9  and  12. 
The  Department  of  Health,  Education,  and  Welfare  reported  in  1977 
that  nationwide  about  one  in  four  students  fail  to  graduate  from 
high  school.  Dropout  rates  for  large  city  schools  may  be  higher. 
One  large  city  reported  that  nearly  half  of  its  students  drop  out. 
In  our  opinion,  the  premature  termination  of  compensatory  education 
services  increases  the  chances  that  underachieving  students  will 
fall  further  behind  their  peers  and  eventually  drop  out  of  school. 
If  this  occurs,  title  I  is  unlikely  to  achieve  its  goals  of  assist¬ 
ing  the  disadvantaged  to  become  self-sufficient  and  of  reducing 
illiteracy  and  poverty. 
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CHAPTER  3 


LACK  OF  EFFECTIVE  TITLE  I  EVALUATIONS 
AND  LIMITED  EXEMPLARY  PROJECT  DATA 
HINDER  ADOPTION  OF  NETWORK  PROJECTS 

Although  many  title  I  projects  need  improvement,  relatively 
few  schools  have  adopted  exemplary  projects.  Only  about  60  (3  per¬ 
cent)  of  the  1, 958  school  districts  in  four  of  the  five  States  we 
visited  had  adopted  an  exemplary  project  for  use  in  their  title  I 
projects  by  the  start  of  the  1978-79  school  year.  (The  Network  fa¬ 
cilitator  for  one  State  said  that  adequate  data  on  project  adop¬ 
tions  were  not  available.)  Only  1  of  the  47  local  school  districts 
we  visited  had  adopted  any  of  the  Network1 s  title  I  exemplary 
projects.  Most  adoptions  of  exemplary  projects  are  for  use  in  a 
single  school ,  thereby  further  limiting  any  impact  the  Network 
could  have  on  title  I  projects. 

Local  school  officials  were  generally  not  receptive  to  adopt¬ 
ing  the  Network' s  exemplary  projects,  in  part  because  inadequate 
evaluations  had  allowed  the  officials  to  remain  satisfied  with 
their  title  I  projects  and  because  they  were  not  convinced  that 
the  exemplary  projects  were  better  than  their  projects.  The 
reluctance  of  school  officials  to  adopt  exemplary  projects  has 
been  perpetuated  by  (1)  the  failure  of  Federal,  State,  and  local 
agencies  to  make  effective  evaluations  and  (2 )  the  lack  of  in¬ 
formation  about  the  exemplary  projects,  which  school  officials 
consider  important  to  convince  them  of  the  projects’  merits. 
Effective  evaluations  and  more  comprehensive  information  about 
exemplary  project  costs  and  achievements  would  help  local  title  I 
officials  to  more  adequately  consider  exemplary  projects  as  an 
alternative . 

However,  regarding  evaluations: 

— ED  has  not  given  States  adequate  criteria  for  assessing 
title  I  project  quality. 

— States  have  not  effectively  assessed  the  quality  of  title  I 
projects,  and  ED  has  not  adequately  monitored  the  States' 
activities  in  this  area. 

— Local  evaluations  of  title  I  projects  have  been  deficient. 

Also,  the  Network's  lack  of  exemplary  project  data  concerning 
student  achievement,  project  costs,  impact  on  student  behavior, 
and  adoption  results  has  contributed  to  local  school  officials' 
lack  of  confidence  in  the  projects. 
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LACK  OF  ADEQUATE  CRITERIA  FOR 
EVALUATING  PROJECT  EFFECTIVENESS 


Title  I  requires  that  local  school  districts  evaluate  the 
effectiveness  of  their  title  I  projects.  Sections  183(b)  and  (d) 
of  title  I  require  ED  to  (1)  develop  and  publish  standards  for 
evaluating  project  effectiveness  and  (2)  provide  models  for 
evaluating  title  I  projects  to  be  used  by  State  education  agen¬ 
cies.  The  models  are  to  include  uniform  procedures  and  criteria 
for  local  and  State  education  agencies  to  use  in  evaluating 
title  I  projects.  Although  this  requirement  has  been  in  effect 
since  1974,  as  of  September  1980,  ED  had  not  provided  adequate 
criteria  for  evaluating  project  effectiveness. 

School  districts  are  required  by  title  I  legislation  to  use 
the  results  of  evaluations  in  planning  for  and  improving  their 
title  I  projects.  Without  more  specific  criteria  to  assess  project 
effectiveness,  local  and  State  education  officials  lack  a  valuable 
tool  for  determining  whether  their  title  I  projects  need  improve¬ 
ment.  One  State  official  said  that,  without  such  specific  criteria, 
local  officials  lack  a  standard  to  use  in  measuring  their  projects' 
effectiveness.  This  official  believed  that  local  school  officials 
would  seek  better  projects  ( including  exemplary  projects)  if  their 
existing  projects  did  not  meet  a  specified  standard. 

In  lieu  of  more  specific  criteria  for  assessing  the  effecive- 
ness  of  title  I  projects,  ED  has  issued  general  instructions  that 
require  local  school  districts  in  making  evaluations  to 

— use  appropriate  and  accurate  tests, 

— use  appropriate  and  accurate  analysis  procedures, 

— include  a  representative  sample  of  the  students  served,  and 

— provide  valid  measures  of  title  I  student  performance  com¬ 
pared  to  estimates  of  what  the  performance  might  have  been 
had  title  I  services  not  been  provided. 

Also,  pursuant  to  requirements  of  title  I  legislation,  ED  has 
prescribed  uniform  evaluation  models  for  school  districts  to  use 
in  assessing  their  projects. 

Although  the  above  instructions  and  models  provide  guidance 
for  evaluation  procedures,  or  inputs,  they  do  not  provide  ade¬ 
quate  criteria  for  determining  what  levels  of  performance  are 
satisfactory. 

ED's  evaluation  models  provide  comparisons  between  the  actual 
gains  of  title  I  students  and  estimates  of  the  gains  that  the 
students  would  have  realized  without  title  I  assistance.  Under 
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this  system,  positive  scores  (gains  above  zero)  indicate  that  the 
title  I  students  performed  above  what  they  would  have  without  com¬ 
pensatory  assistance.  Negative  scores  indicate  that  the  students' 
performance  was  below  what  they  would  have  realized  without  such 
assistance.  ED  guidelines,  however,  do  not  specify  what  levels 
of  positive  gain  might  be  considered  appropriate,  or  what  levels 
might  indicate  project  improvements  were  needed. 

ED  officials  told  us  that  the  specification  of  criteria  to 
separate  satisfactory  from  unsatisfactory  projects  is  technically 
unsound  and  educationally  indefensible.  ED  officials  believe  a 
requirement  that  performance  standards  be  met  often  results  in  dis¬ 
tortions  and  dishonesty  in  evaluations  and  fosters  an  unnecessary, 
counterproductive  adversarial  atmosphere  between  agencies. 

ED  NEEDS  TO  IMPROVE  MONITORING 
OF  STATE  TITLE  I  ACTIVITIES 

ED  is  responsible  for  assuring  that  State  education  agencies 
are  meeting  their  responsibilities  under  the  title  I  program. 

State  agencies  have  primary  responsibility  for  monitoring  and 
evaluating  local  title  I  projects.  As  part  of  this  responsibility 
they  are  to  encourage  schools  with  deficient  title  I  projects  to 
adopt  promising  educational  programs.  However,  neither  ED  nor 
States  have  met  their  responsibilities.  Consequently,  they  are 
not  always  aware  of  projects  that  need  improvement  and,  therefore, 
are  not  able  to  encourage  adoptions  of  exemplary  projects  or  to 
provide  other  guidance  for  improving  title  I  projects. 

ED  monitoring  of  State  title  I 
monitoring  is  limited 

Although  ED  is  responsible  for  determining  whether  States  are 
meeting  their  title  I  responsibilities,  it  does  not  direct  its  ac¬ 
tivities  toward  assuring  that  States  foster  effective  title  I 
projects.  ED  is  required  by  title  I  legislation  to: 

— Assess  whether  State  applications  meet  the  requirements  of 
title  1  law  and  whether  the  requirements  will  be  carried 
out . 


— Submit  enforcement  reports  to  the  Congress  disclosing  the 
(1 )  extent  to  which  State  procedures  satisfy  the  title  I 
requirements,  (2)  manner  in  which  ED's  monitoring  reports 
were  considered  in  approving  State  applications,  (3)  find¬ 
ings  of  ED's  onsite  monitoring  visits,  and  (4)  actions  taken 
to  correct  problems  identified  during  the  visits. 

ED's  administration  of  title  I  has  not  been  effective  in  get¬ 
ting  States  to  meet  program  development  responsibilities.  Histori¬ 
cally,  ED's  title  I  monitoring  has  focused  on  whether  the  funds 


were  used  properly,  not  on  whether  they  were  used  effectively. 
Consequently,  most  of  the  weaknesses  cited  by  ED  in  its  reviews 
of  State  title  I  activities  have  pertained  to  violations  of  the 
technical  provisions  of  the  law  and  not  to  program  effectiveness. 

ED  officials  told  us  that  their  title  I  monitoring  emphasized 
compliance  with  appropriate  procedures  and  not  achievement  or 
quality.  Our  review  of  14  ED  monitoring  reports  for  the  five  States 
we  visited  disclosed  that  virtually  all  the  problems  identified  by 
ED  monitoring  dealt  with  compliance  violations.  None  of  the  14 
monitoring  reports  we  reviewed  contained  exceptions  relating  to 
project  accomplishment  or  disclosed  whether  project  achievement 
levels  were  adequate.  States  also  directed  their  monitoring  ef¬ 
forts  toward  compliance  elements. 

In  the  past,  insufficient  staff  levels  may  have  hampered  ED's 
ability  to  adequately  monitor  title  1  projects.  In  its  1978  Annual 
Report,  the  National  Advisory  Council  on  the  Education  of  Dis¬ 
advantaged  Children  stated  that  ED' s  title  I  monitoring  had  been 
substantially  reduced.  The  National  Institute  of  Education' s  Com¬ 
pensatory  Education  Study  1/  also  expressed  concern  over  ED' s 
ability  to  adequately  monitor  State  title  I  activities.  The  In¬ 
stitute  reported  that  ED's  monitoring  and  enforcement  had  declined 
significantly  and  that  States  were  not  effectively  supervised. 

ED  officials  said  that,  after  issuance  of  the  Council  and 
Institute  reports,  ED  increased  its  title  I  monitoring  staff  from 
18  to  37  persons.  They  said  that  ED  intends  to  put  more  emphasis 
on  monitoring  title  I  project  achievement. 

State  assessments  of  project 
effectiveness  are  not  adequate 

States  are  required  by  law  to  assess  local  title  I  projects 
to  assure  that  (1)  project  quality  is  acceptable,  (2)  local  evalua¬ 
tions  are  properly  conducted,  and  (3 )  program  deficiencies  are  cor¬ 
rected.  State  title  I  assessments,  however,  generally  do  not  focus 
on  project  quality  and  are  not  adequate  to  detect  title  I  projects 
needing  improvement. 

State  education  agencies  have  primary  responsibility  for  moni¬ 
toring  local  title  I  projects.  In  enacting  title  I,  the  Congress 
considered  States'  monitoring  essential  to  determining  the  quality 
of  title  I  services  at  the  local  level.  States'  responsibilities, 
as  set  forth  in  titles  I  and  V  of  ESEA,  are  to 


^/"Compensatory  Education  Study:  Administration  of  Compensatory 
Education,"  National  Institute  of  Education,  1977. 
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— monitor  the  effectiveness  of  title  I  projects 


— determine  whether  local  title  I  projects  are  of  sufficient 
quality, 

— determine  whether  local  school  districts  are  adequately 
evaluating  the  effectiveness  of  their  title  I  projects, 

— evaluate  the  effectiveness  of  title  I  projects  in  improv¬ 
ing  the  educational  attainment  of  educationally  disadvan¬ 
taged  children, 

— assure  that  deficiencies  and  noncompliant  practices  detected 
through  monitoring  and  evaluation  are  corrected,  and 

— disseminate  and  encourage  adoption  of  promising  educational 
practices . 

State  agency  monitoring  activities,  however,  generally  focus 
on  school  districts'  "compliance"  with  title  I  requirements  dealing 
with  such  matters  as  allocating  funds  and  selecting  children  to  be 
served.  Little  State  monitoring  is  directed  toward  program  develop¬ 
ment  or  improvement. 

A  1977  study  of  title  I  administration  1/  funded  by  the  De¬ 
partment  of  Health,  Education,  and  Welfare  reported  that,  although 
States  are  responsible  for  helping  local  school  districts  to  im¬ 
prove  program  operations,  many  States  had  not  assumed  their  program 
development  responsibility.  The  study  stated  that: 

— State  monitoring  of  program  quality  was  superficial  and 
historically  incomplete. 

— States  did  not  have  adequate  systems  for  title  I  project 
development. 

— States  did  not  hold  school  districts  accountable  for  their 
title  I  programs  and  were  not  doing  an  adequate  job  of  get¬ 
ting  school  districts  to  implement  effective  title  I 
services . 

— State  title  I  evaluation  policy  was  restricted  to  ensuring 
that  local  test  data  had  been  submitted . 

— States  did  not  question  the  uses  local  school  districts 
were  making  of  their  evaluations  and  made  little  effort 
to  help  schools  link  project  evaluations  to  project  design. 


1/ "A  Study  of  the  Administration  of  the  Elementary  and  Secondary 
Education  Act,  Title  I,  in  Eight  States,"  1977. 
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— States  had  not  informed  school  districts  that  evaluations 
are  important  to  project  design . 

— States  implied  that  they  did  not  attach  much  importance  to 
program  improvement  when  they  failed  to  examine  how  evalua¬ 
tions  are  used  to  plan  for  title  I  activities. 

State  monitoring  in  all  five  State  education  agencies  we 
visited  focused  on  compliance  elements,  not  project  effectiveness. 
Title  1  officials  at  one  State  agency  told  us  that  more  extensive 
monitoring  was  needed  to  enable  the  State  to  identify  project  weak¬ 
nesses  and  encourage  schools  to  adopt  exemplary  projects.  These 
officials  said  their  State  lacked  title  I  project  evaluation  data 
because  it  had  a  small  monitoring  staff.  Only  one-fourth  of  a 
full-time  equivalent  staff  person  was  budgeted  for  project  monitor¬ 
ing  and  evaluation. 

The  title  I  coordinator  in  another  State  said  that  it  is  not 
the  State  monitors'  function  to  recommend  program  changes  when 
weaknesses  are  observed.  The  State's  function,  according  to  this 
official,  is  to  monitor  for  compliance  with  the  title  I  regula¬ 
tions,  not  to  recommend  changes  or  encourage  schools  to  adopt 
exemplary  projects. 

Public  Law  95-561,  enacted  November  1,  1978,  clarified  the 
role  of  States  in  title  I  program  management.  In  developing 
title  I  legislation,  the  Congress  intended  that  States  consider 
local  project  evaluations  before  approving  local  applications  for 
title  I  funds.  States  were  expected  to  disapprove  applications 
for  projects  that  had  not  raised  levels  of  student  achievement. 

Local  title  I  officials  tend  to  be  reluctant  to  implement 
changes  to  their  projects.  Stimulus  from  State  education  offi¬ 
cials  might  be  necessary  to  alert  school  officials  to  project 
inadequacies  and  to  motivate  project  improvements.  States  that 
lack  suitable  assessments  of  project  quality  are  not  able  to  pro¬ 
vide  such  stimulus  and  are  not  meeting  the  intent  of  title  I  leg¬ 
islation. 

LOCAL  PROJECT  EVALUATIONS  ARE  DEFICIENT 

Local  school  districts  are  required  to  evaluate  the  effec¬ 
tiveness  of  their  title  I  projects  and  to  report  the  evaluation 
results  to  State  education  agencies.  Specifically,  title  I  leg¬ 
islation  requires  that  local  school  districts: 

— Adopt  effective  procedures  for  evaluating  their  title  I 
projects. 

— Use  objective  measures  of  educational  achievement. 
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— Use  the  results  of  the  school  districts'  evaluations  in 
planning  for  and  improving  title  I  project  activities. 

Local  school  district  evaluations  varied  considerably  in  na¬ 
ture  and  quality.  Generally,  however,  these  evaluations  included 
various  forms  of  tests  to  measure  student  achievement  levels.  The 
local  evaluations  were  usually  based  on  standardized  norm- 
referenced  tests.  In  the  past,  these  tests  were  designed  to  meas¬ 
ure  the  achievement  level  of  students  in  relation  to  the  level 
attained  by  a  nationally  representative  student  sample.  The  norm 
scores  were  based  on  the  results  of  tests  given  by  the  testmaker 
to  samples  of  students  intended  to  be  representative  of  the  group 
for  whom  the  tests  were  designed.  As  indicated  on  page  18,  ED's 
new  evaluation  models  provide  for  comparisons  between  the  stu¬ 
dents'  achievement  and  estimates  of  what  the  students  would  have 
achieved  without  title  I  assistance. 

School  district  evaluations,  however,  are  seldom  used  to  iden¬ 
tify  needed  program  improvements  and  were  often  based  on  method¬ 
ologies  that  produce  inaccurate  results.  School  officials' 
limited  adoption  of  the  Network' s  exemplary  projects  may  have 
stemmed  in  part  from  deficiencies  in  local  school  district  evalua¬ 
tions  of  title  I  activities.  Superficial  or  deficient  project 
evaluations  may  fail  to  disclose  serious  project  shortcomings. 
Unaware  of  these  shortcomings,  local  officials  may  see  no  reason 
to  adopt  the  Network's  exemplary  projects.  Despite  the  lack  of 
specific  criteria  for  measuring  success,  implementing  ED's  evalua¬ 
tion  models  could  improve  the  quality  of  local  evaluation  models. 

Local  evaluations  not  used 
for  project  improvement 

In  designing  title  I  legislation,  the  Congress  intended  that 
local  evaluations  would  enable  school  districts  to  identify  weak¬ 
nesses  in  their  title  I  projects  and  would  serve  as  a  tool  for 
project  revision  and  improvement.  Local  title  I  officials,  how¬ 
ever,  tend  to  resist  change  to  their  projects  even  though  some 
project  evaluations  showed  (1)  many  of  the  students  were  continu¬ 
ing  to  fall  behind,  (2)  many  needy  students  were  totally  excluded 
from  the  projects,  and  (3)  many  students  were  dropped  from  the 
projects  before  reaching  the  level  of  achievement  of  their  peers. 
Officials  in  41  of  the  47  districts  we  visited  were  not  interested 
in  adopting  new  title  I  projects.  Most  were  satisfied  with  their 
existing  projects  and  therefore  not  willing  to  consider  replacing 
them. 

Studies  funded  by  the  Department  of  Health,  Education,  and 
Welfare  have  reported  that  school  districts  made  little  use  of 
title  I  evaluations  for  project  improvement.  Instead,  local 
evaluations  were  used  primarily  to  meet  title  I  reporting 
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requirements.  The  1977  study  of  the  administration  of  title  I  1/ 
concluded  that  local  school  districts 

— lacked  systematic  processes  for  effectively  developing 
title  I  projects, 

— gave  little  attention  to  project  planning  aspects  of 
their  evaluations,  and 

— prepared  evaluations  primarily  to  meet  reporting  require¬ 
ments  . 

Another  study  2/  dealing  with  the  uses  of  local  title  I  evalua¬ 
tions  reported  that,  rather  than  critically  assessing  their  projects, 
school  officials  tended  to  seek  evidence  supporting  their  positive 
feelings  and  to  ignore  evidence  to  the  contrary. 

Inappropriate  methodology  reduces 
the  validity  of  local  evaluations 

Deficiencies  in  local  assessments  of  project  effectiveness 
have  been  compounded  by  the  low  quality  of  title  I  evaluations. 

We  previously  discussed  this  problem,  along  with  the  attitude  of 
local  officials  that  title  I  evaluations  are  not  useful  for  assess¬ 
ing  project  effectiveness,  in  a  September  1977  report.  3/  In  that 
report,  we  pointed  out  that  over  half  of  the  State  title  I  offi¬ 
cials  we  surveyed  believed  that  local  title  I  evaluations  were 
less  than  adequate  in  terms  of  credibility  of  findings,  presen¬ 
tation  of  required  management  information,  and  qualification  and 
quantification  of  measurement  data.  Also,  our  report  said  that 
studies  by  other  organizations  had  shown  that: 

— Evaluation  designs  were  not  adequate  to  produce  reliable 
data  on  measurable  achievement  gains. 

— School  officials  lacked  incentives  to  collect  or  report 
program  output  data . 

— School  personnel  did  not  show  much  ability  or  interest  in 
using  evaluations  to  formulate  title  I  policy  or  practice. 


_l/lbid,  page  27. 

2/"Local  Uses  of  Title  I  Evaluations,"  SRI  International,  1978. 

3/ "Problems  and  Needed  Improvements  in  Evaluating  Office  of  Educa¬ 
tion  Programs"  (HRD-76-16  5,  Sept.  8,  19  77). 
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Problems  were  also  noted  in  the  way  local  school  districts 
conducted  their  project  evaluations  during  our  review  at  14  school 
districts.  These  problems  seriously  affect  the  reliability  and 
usability  of  the  evaluations  to  determine  whether  and  how  projects 
need  to  be  improved.  Some  of  these  problems  are  the  (1)  failure 
to  control  sample  groups  in  evaluating  project  achievement,  (2) 
use  of  improper  methods  to  compute  student  achievement,  (3)  use 
of  inappropriate  tests,  (4)  failure  to  measure  sustained  gain  and 
total  program  effect,  and  (5)  failure  to  consider  project  ef¬ 
ficiency  and  cost  effectiveness. 

Sample  groups  not  controlled 

Achievement  in  title  I  projects  is  generally  measured  by  pre¬ 
testing  students  near  the  start  of  a  school  year  and  posttesting 
them  near  the  end.  The  difference  in  average  achievement  level 
between  the  students  pretested  and  those  posttested  is  considered 
the  average  gain  for  the  subject  and  grade  level  being  evaluated. 

For  such  a  measurement  to  be  meaningful,  the  sample  groups  tested 
must  be  carefully  controlled  to  assure  that  the  evaluations  measure 
the  intended  characteristics  accurately. 

However,  some  school  districts  did  not  control  their  test 
groups  to  assure  that  students  tested  actually  received  title  I 
assistance  in  the  subjects  being  evaluated.  As  a  result,  some 
evaluations  measured  the  gains  of  students  who  were  not  receiving 
title  I  assistance  for  the  subject  evaluated.  Many  such  students 
were  scoring  well  above  the  national  average  on  the  tests,  thereby 
possibly  biasing  the  evaluation  results.  Where  this  occurs,  the 
evaluations  do  not  provide  accurate  information  on  the  achievement 
gain  of  students  actually  served  and,  therefore,  are  not  useful  in 
determining  whether  improvements  are  needed . 

A  related  problem  was  the  failure  to  assure  that  the  same  stu¬ 
dents  were  pretested  and  posttested.  Obviously,  testing  one  student 
at  the  beginning  of  the  year  and  another  at  the  end  will  not  yield 
information  on  the  gain  of  either  student.  However,  it  was  not 
unusual  for  as  many  as  40  percent  of  the  students  to  be  either  pre¬ 
tested  but  not  posttested,  or  vice  versa.  Some  school  districts 
do  not  consider  differences  in  the  makeup  of  students  between  pre¬ 
test  and  posttest  groups.  Such  differences  in  the  test  groups  re¬ 
sulted  in  title  I  project  achievement  gains  being  substantially 
overstated  or  understated.  ED  officials  said  that  guidelines  for 
project  evaluation  now  specify  that  pretesting  and  posttesting 
must  be  compared  on  an  individual  student  basis  and  that,  if  in¬ 
structions  are  followed  correctly,  pretest  and  posttest  groups 
should  be  the  same . 
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Improper  methods  used  to 

compute  student  achievement 

Some  school  systems '  procedures  for  computing  student  achieve¬ 
ment  produced  distorted  measurements.  In  several  school  districts, 
for  example,  title  I  evaluators  were  using  methods  to  compile  aver¬ 
age  grade  level  equivalent  scores  that  distort  computations  of 
achievement  levels. 

In  another  district  most  of  the  second  grade  students  were 
not  pretested,  and  gain  was  measured  as  if  the  students  had  started 
from  a  pretest  score  of  zero.  This  practice  improperly  inflated 
test  results  because  gains  achieved  over  a  2-year  period  were  re¬ 
ported  as  a  1-year  gain. 

Inappropriate  tests  used 

Some  schools'  inappropriate  use  of  achievement  tests  produced 
misleading  or  useless  results.  In  some  cases,  for  example,  the 
posttests  and  pretests  measured  different  skills.  Some  schools 
used  tests  that  were  not  appropriate  for  the  achievement  levels  of 
the  students  tested. 

Some  schools  frequently  changed  the  brand  or  type  of  test  used 
to  measure  student  achievement.  At  times,  the  change  was  made 
within  a  single  evaluation  period.  Frequent  changes  in  types  of 
tests  make  it  difficult  to  compare  and  evaluate  the  test  results. 

One  district  had  developed  its  own  tests  to  measure  skills 
taught  in  the  classroom.  The  title  I  evaluation  system  allows  for 
the  use  of  locally  developed  tests  where  the  gain  on  the  local  test 
is  translated  into  a  national  metric  or  scale.  However,  if  the 
gain  on  the  local  test  is  inaccurate,  translations  to  a  national 
metric  or  scale  will  not  correct  the  problem.  Analysis  of  achieve¬ 
ment  tests  by  testing  consultants  showed  that  most  locally  devel¬ 
oped  tests  were  of  poor  quality.  Locally  developed  tests  are  not 
thoroughly  tested,  as  are  most  of  the  standardized  achievement 
tests  more  commonly  used  for  program  evaluation. 

Sustained  gain  and  total 

program  effect  not  measured 

Although  most  students  are  dropped  from  title  I  in  the  early 
grades  and  before  they  have  reached  the  average  achievement  level 
of  their  classmates,  few  school  districts  have  attempted  to  follow 
the  students’  progress  to  see  whether  their  title  I  experience  has 
any  sustaining  effect.  School  officials  tend  to  view  and  manage 
their  title  I  programs  as  annual  efforts,  rather  than  as  a  series 
of  integrated  units  designed  for  cumulative  impact  on  the  students’ 
academic  success.  None  of  the  districts  in  our  review  had  analyzed 
the  cumulative  effect  of  consecutive  years  of  title  I  assistance 
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on  the  rate  of  student  achievement  gain.  Similarly,  the  districts 
had  not  attempted  to  follow  students  after  they  were  released  from 
title  I  to  see  whether  the  projects  had  any  lasting  effect  on  stu¬ 
dent  achievement,  dropout  rates,  attitudes  toward  school,  or  other 
factors. 

Some  impetus  has  been  given  to  the  measurement  of  sustained 
gain  by  the  Education  Amendments  of  1978.  ESEA  title  I  now  re¬ 
quires  local  project  evaluations  to  include  measurements  of 
achievement  over  at  least  a  12-month  period  at  least  once  during 
each  3-year  period.  School  districts  are  required  to  use  this 
information  by  considering  the  inclusion  of  project  components 
designed  to  sustain  student  achievement  beyond  the  school  year  in 
which  the  projects  are  conducted. 

Program  efficiency  and  cost 

effectiveness  not  considered 

Only  one  school  district  we  visited  had  attempted  to  analyze 
the  cost  effectiveness  of  the  various  title  I  projects  offered  in 
its  schools.  Education  officials  testified  that  funding  limita¬ 
tions  have  caused  school  districts  to  exclude  many  eligible  stu¬ 
dents  or  terminate  their  title  I  services.  Measures  of  cost 
effectiveness  could  help  school  officials  determine  whether  their 
projects  are  efficient  in  comparison  with  other  schools  and  dis¬ 
tricts  or  with  exemplary  projects. 

Agency  actions  to  improve  evaluations 

ED  has  focused  substantial  effort  on  improving  title  I 
evaluations.  In  compliance  with  a  requirement  originally  added 
to  ESEA  in  1974,  section  151  of  Public  Law  93-380,  ED  developed 
evaluation  models  and  standards  for  use  by  State  and  local  educa¬ 
tional  agencies.  On  October  12,  1979,  final  regulations  were  pub¬ 
lished  specifying  models  for  evaluating  the  effectiveness  of  title  I 
projects  providing  instructional  services  in  reading,  language  arts, 
or  mathematics.  ED's  three  evaluation  models  were  developed  to 
provide  reliable  data  on  program  accomplishments.  The  models  were 
also  designed  to  permit  the  aggregation  and  comparison  of  project 
results,  even  though  the  school  districts  use  different  kinds  of 
tests . 


In  1976,  10  regional  Technical  Assistance  Centers  were  estab¬ 
lished  under  contract  to  ED  to  provide  evaluation  technical  assist¬ 
ance  when  requested  by  States  and  their  local  school  districts. 

The  Centers  were  established  to  assist  States  and,  at  the  discre¬ 
tion  of  the  States,  local  districts  in  implementing  the  title  I 
Evaluation  and  Reporting  System  and  in  dealing  with  other  title  I 
matters.  The  Centers'  function  is  to  improve,  through  training 
and  consultation.  State  and  local  capability  for  performing  title  I 
evaluation.  Technical  assistance  focuses  attention  on  the  evalua¬ 
tion  models  and  reporting  forms  included  in  the  title  I  evaluation 
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system.  Training  consists  primarily  of  evaluation  workshops 
requested  by  the  States,  intermediate  service  units,  or  school 
districts.  Consultation  is  provided  to  facilitate  the  actual 
implementation  of  one  or  more  of  the  title  I  evaluation  system 
components  in  site  specific  situations.  The  Centers  also  provide 
assistance  in  data  utilization  and  quality  control ,  such  as  adher¬ 
ence  to  testing  dates,  accuracy  in  completing  forms,  score  conver¬ 
sions,  data  analysis,  and  data  aggregation. 

NETWORK  DATA  ON  EXEMPLARY 
PROJECTS  NEED  IMPROVEMENT 


Readily  available  information  on  Network  exemplary  projects 
often  does  not  show  the  projects'  effectiveness  based  on  (1)  the 
developers'  experience  or  (2 )  the  experiences  of  schools  that  have 
adopted  the  projects.  The  absence  of  convincing  data  on  the  merits 
of  exemplary  projects  has  contributed  to  school  officials’  opposi¬ 
tion  to  adopting  exemplary  projects. 

Moreover,  the  impact  of  the  Network  on  title  I  projects  may 
be  overstated  because  the  Network' s  accounting  for  the  number  of 
adoptions  for  title  I  projects  includes  adoptions  that  did  not  take 
place  and  adoptions  for  other  than  title  I  projects. 

Data  on  the  merits  of  exemplary 
projects  inadequate 

A  brief  summary  description  of  each  Network  project  is  in¬ 
cluded  in  a  catalog  entitled  "Educational  Programs  That  Work." 

The  catalog,  issued  annually  by  the  Network,  includes  the  name  and 
telephone  number  of  the  project  director ,  who  can  be  contacted  for 
further  information.  However,  ED  officials  stated  that  their  ex¬ 
perience  with  the  Network  was  that  few  decisions  to  adopt  a  Network 
project  were  made  as  a  result  of  reading  printed  materials.  Ac¬ 
cording  to  ED,  a  study  found  that  most  decisions  to  adopt  an  exem¬ 
plary  project  are  made  as  a  result  of  representatives  from  a  school 
district  either  visiting  a  project  to  see  it  in  operation  or  at¬ 
tending  an  awareness  conference  and  talking  with  the  project' s 
developer . 

Most  school  officials  we  interviewed  believed  that  the  Net¬ 
work'  s  exemplary  projects  were  not  superior  to  their  school’ s 
title  I  projects.  Available  Network  literature  about  the  exemplary 
projects  frequently  lacked  convincing  data  to  establish  their  super¬ 
iority.  Without  such  data  it  is  questionable  whether  school  offi¬ 
cials  who  were  not  already  interested  in  a  project  would  visit  the 
project  or  attend  an  awareness  conference  concerning  it.  Although 
project  developers  provide  information  demonstrating  their  projects' 
effectiveness  to  the  Joint  Dissemination  Review  Panel,  that  informa¬ 
tion  is  not  usually  made  available  to  prospective  project  adopters. 
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Our  review  of  the  catalogs  and  other  Network  literature  showed  that 


— data  on  sustained  achievement  gain  were  available  for  only 
1  of  the  21  exemplary  title  I  projects, 

—  literature  for  6  projects  was  devoid  of  any  cost- 
effectiveness  data,  and 

— information  on  student  behavior  and  attendance  was  not 
available  for  18  of  the  21  projects. 

Local  title  I  officials  at  34  school  districts  told  us  that 
one  reason  they  had  not  adopted  exemplary  title  I  projects  was 
that  they  could  not  evaluate  whether  the  projects  were  suitable 
for  their  districts.  Officials  of  13  of  the  districts  told  us 
that  the  literature  distributed  by  the  Network  and  the  developer- 
demonstrators  on  exemplary  projects  was  not  convincing .  For  ex¬ 
ample,  officials  of  one  district  said  they  considered  adopting  an 
exemplary  title  I  project  but  decided  against  it  because  the 
limited  information  available  about  project  results  was  insuffi¬ 
cient  to  convince  them  that  the  project  would  be  better  than  what 
they  had.  District  officials  hoped  to  see  a  record  of  the  exem¬ 
plary  projects'  effectiveness  over  a  period  of  years. 

Even  some  of  the  school  districts  that  adopted  exemplary 
projects  complained  about  the  adequacy  of  information  concerning 
the  projects.  Officials  at  several  of  these  districts  told  us  that 
information  they  desired  to  see  on  the  long-term  effects  of  the  ex¬ 
emplary  projects  on  student  learning  performance  was  not  available. 
One  official  stated  that  the  limited  information  that  was  available 
before  a  project  was  adopted  had  made  it  difficult  to  convince 
school  staff  and  officials  to  adopt  the  project. 

Officials  of  another  school  that  adopted  an  exemplary  project 
said  detailed  information  on  projects  had  been  difficult  to  obtain. 
They  said  that  the  information  they  were  able  to  get  on  project 
effect,  sustained  growth  and  attitude,  attendance,  and  behavior 
improvement  was  mostly  unsubstantiated  opinion. 

Also,  some  State  facilitators  told  us  that,  because  of  the 
lack  of  project  data,  they  were  not  able  to  respond  to  school  of¬ 
ficials'  questions  about  exemplary  projects.  Various  State  fa¬ 
cilitator  officials  expressed  concerns  over  the  lack  of 

— data  to  determine  which  projects  were  best  suited  to 
particular  school  districts, 

— data  on  the  cost  of  operating  some  projects, 

— data  that  would  enable  school  officials  to  make  project 
comparisons , 
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— data  on  project  implementation  requirements  that  would 
enable  officials  to  assess  their  schools'  abilities  to 
implement  the  projects,  and 

—  current  data  on  projects. 

Regarding  the  currency  of  data,  our  review  of  project  litera¬ 
ture  that  was  dated  disclosed  that  about  one- fourth  of  the  projects 
were  presenting  data  4  or  more  years  old.  In  one  case,  student 
achievement  data  were  11  years  old.  ED  officials  told  us  that  they 
generally  did  not  obtain  performance  data  for  the  exemplary  proj¬ 
ects  after  the  year  the  projects  were  certified  by  the  Joint  Dis¬ 
semination  Review  Panel.  Nineteen  of  the  title  I  exemplary  proj¬ 
ects  had  been  presented  to  the  panel  before  1975.  Consequently, 
current  data  had  not  been  available  to  ED  or  potential  adopters 
to  demonstrate  that  the  current  project  formats  were  producing  the 
same  exemplary  results. 

Feedback  on  effectiveness  of 
adopted  projects  not  obtained 

The  Network  has  not  routinely  obtained  information  on  the  re¬ 
sults  achieved  by  the  districts  or  schools  that  adopted  exemplary 
projects.  In  our  opinion,  such  information  would  be  useful  to  the 
Network  in  promoting  adoptions  of  exemplary  projects  by  demonstrat¬ 
ing  that  exemplary  achievements  can  be  replicated  at  adoption  sites. 

Although  a  major  purpose  of  the  Network  is  to  encourage  im¬ 
provements  through  the  adoption  of  exemplary  projects,  virtually 
the  only  information  the  Network  had  on  past  adoptions  was  a  list 
of  them.  At  the  time  of  our  fieldwork,  the  Network  did  not  have 
a  formal  system  to  acquire  information  on  the  success  or  failure 
of  title  I  project  adoptions. 

The  Network  director  told  us  that  the  Network' s  efforts  fo¬ 
cused  on  getting  schools  to  adopt  exemplary  projects.  Followup  ac¬ 
tivities  were  not  considered  a  high  priority,  and  little  followup 
had  been  done.  Officials  of  five  State  facilitators  and  six 
developer- demonstrators  we  interviewed  indicated  that  they  gen¬ 
erally  did  not  follow  up  to  see  what  happened  after  a  district  or 
school  agreed  to  adopt  an  exemplary  project. 

Our  interviews  with  the  five  State  facilitators  disclosed 

that 


— none  could  provide  information  on  how  adopted  exemplary 
projects  had  affected  student  learning  and  achievement 
levels ; 

— four  did  not  know  whether  some  of  the  reported  adoptions 
had  been  implemented? 
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— three  believed  a  better  followup  system  was  needed  to 

evaluate  the  results  of  past  adoptions,  but  said  that  fol¬ 
lowup  efforts  on  adoptions  would  require  greater  funding 
for  additional  staff ; 

— one  said  that  with  additional  funds  he  could  make  evalua¬ 
tions  of  project  quality,  which  are  not  now  provided  in  the 
contract;  and 

— one  said  that,  because  his  office  has  only  two  professionals, 
not  much  followup  can  be  provided. 

Most  of  the  seven  developer- demonstrators  we  interviewed  also 
cited  a  need  for  better  followup  to  evaluate  the  effects  of  adop¬ 
tions  and  to  assist  in  adoptions.  However,  only  one  developer- 
demonstrator  had  followed  up  to  obtain  data  on  the  accomplishments 
of  adopted  projects.  The  Network  published  this  information  to 
publicize  the  impact  of  the  adoptions.  Officials  of  the  other 
five  developer- demonstrators  indicated  a  serious  lack  of  informa¬ 
tion  about  the  results  of  prior  adoptions.  In  this  regards 

— Officials  of  four  developer-demonstrators  said  they  performed 
little  or  no  followup  and  could  not  tell  us  how  many  adop¬ 
tions  of  their  projects  were  still  in  effect.  Three  of  them 
were  not  sure  whether  all  of  their  claimed  adoptions  had 
been  implemented.  In  contacting  15  of  the  school  districts 
that  a  developer- demonstrator  reported  had  adopted  projects, 
we  found  that  4  had  never  adopted  the  project  and  2  had 
dropped  the  project.  Officials  of  the  developer- demonstrator 
were  not  aware  of  these  six  cases  or  the  reasons  the  projects 
were  not  operational . 

— An  official  of  another  developer- demonstrator  told  us  that 
it  asked  all  school  districts  adopting  its  project  to  provide 
student  achievement  data  for  use  in  evaluating  the  success 
of  adoptions.  However,  only  3  of  over  140  adopting  districts 
submitted  the  requested  data.  The  official  stated  that  the 
developer- demonstrator  depends  on  the  adopting  schools  to 
submit  information  voluntarily,  since  it  has  neither  the 
time  nor  the  staff  to  follow  up.  Unless  the  adopting  schools 
contact  it,  the  developer- demonstrator  does  not  know  if  the 
project  was  dropped  or  even  adopted. 

Because  of  the  importance  of  feedback  data  on  project  adop¬ 
tions,  some  developer- demonstrators  said  the  Network  should  provide 
greater  emphasis  and  increased  funding  for  periodic  followup  and 
for  the  evaluation  of  adoption  results. 

Several  school  district  officials  stated  that  the  Network' s 
lack  of  assistance  for  followup  activities  has  adversely  affected 
their  adopted  projects.  According  to  these  officials,  the 


31 


Network' s  training  and  followup  efforts  were  inadequate  when  pro¬ 
ceeding  through  critical  stages  of  the  project.  An  official  of 
one  school  district  said  that,  if  the  Network  wants  to  keep  the 
adopted  projects  from  being  diluted  and  keep  the  "momentum  going," 
the  personnel  of  school  districts  adopting  exemplary  p:oje<_ts 
should  be  retrained  periodically.  Officials  of  two  other  school 
districts  that  had  operated  exemplary  projects  for  2  years  without 
any  support  from  the  developers  said  they  would  like  postadoption 
assistance,  including  access  to  material,  techniques,  and  feedback 
from  the  developer . 

The  need  for  careful,  thorough  implementation  of  exemplary 
projects  was  emphasized  by  a  4-year  ED-funded  study  of  Federal  pro¬ 
grams  supporting  educational  change.  _1/  Although  some  successful 
replications  were  observed,  the  study  concluded  that 

— successful  projects  were  not  disseminated  easily, 

—  replication  at  new  sites  usually  fell  short  of  performance 
at  the  original  sites, 

— few  projects  were  successfully  implemented,  and 

—  fewer  survived  in  the  long  run. 

Based  on  an  analysis  of  the  factors  influencing  the  success 
of  the  adoption  processes,  the  study  reported 

— implementing  strategies  can  make  the  difference  between 
success  or  failure, 

— implementing  strategies  can  determine  whether  teachers 
would  assimilate  and  continue  using  project  methods  or 
allow  them  to  fall  into  disuse, 

— one-shot  preimplementation  training  is  ineffective,  and 

— extended  training  and  classroom  assistance  from  project 
staff  are  effective  implementation  strategies. 

The  study  concluded  that  the  inability  of  many  school  districts 
to  implement  and  sustain  program  change  ultimately  frustrates  the 
objectives  of  Federal  education  programs.  Instead  of  concentrating 
only  on  the  initial  stage  of  the  adoption  process,  the  study  recom¬ 
mended  that  the  Federal  role  be  expanded  to  subsequent  adoption 
stages  in  order  to  assure  the  success  and  long-term  institution¬ 
alization  of  adopted  projects. 


_l/"Federal  Programs  Supporting  Educational  Change,  Vol.  VIII: 
Implementing  and  Sustaining  Innovations,"  RAND  Corporation, 
1978. 
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Better  information  needed  to  assess 
the  Network1 8  impact  on  title  I 

Accurate  information  on  the  number  of  schools  adopting  Network 
title  I  projects  would  provide  a  more  reasonable  basis  for  deter¬ 
mining  (1)  the  extent  to  which  the  Network's  projects  are  being 
used  and  (2)  the  need  for  any  adjustments  to  increase  the  Network's 
potential  for  improving  title  I  projects. 

From  its  inception  in  1974  through  the  fall  of  1978,  the  Net¬ 
work  approved  and  funded  for  dissemination  21  exemplary  title  1 
projects.  The  Network,  however,  has  not  determined  how  many  dis¬ 
tricts  and  schools  have  adopted  the  21  projects  for  title  I 
projects.  The  Network  received  reports  on  adoptions  of  exemplary 
projects  from  its  developer- demonstrators ,  but  these  reports  did 
not  provide  an  accurate  basis  for  recording  title  I  adoptions. 

Developer- demonstrators  frequently  reported  only  the  number 
of  districts  that  adopted  projects  without  indicating  the  number 
of  schools  adopting  in  each  district?  therefore,  the  Network  did 
not  know  how  many  schools  adopted  a  project  in  each  reported  adopt¬ 
ing  district.  According  to  ED,  most  adoptions  of  exemplary  projects 
are  not  for  use  in  schools  throughout  the  district  and  often  are 
only  for  use  in  a  single  school.  Reporting  on  a  district  basis 
tends  to  make  the  Network' s  reported  accomplishments  appear  more 
extensive  than  actually  is  the  case. 

Also,  developer- demonstrator  reports  did  not  provide  assur¬ 
ance  that  reported  adoptions  actually  occurred  and  were  continued. 
Our  followup  on  36  adoptions  recorded  by  State  officials  showed 
that  5  had  not  occurred  and  4  had  been  discontinued.  The  Network 
did  not  adequately  follow  up  to  verify  that  districts  and  schools 
had  adopted  and  continued  to  use  exemplary  projects  as  reported. 

Moreover,  as  previously  stated,  reported  adoptions  of  title  I 
exemplary  projects  include  adoptions  for  use  in  other  than  title  I 
activities.  Our  followup  of  87  adoptions  that  occurred  in  four 
States  we  visited  showed  that  at  least  59  were  not  for  use  in 
title  I  projects.  While  we  do  not  question  the  desirability  of 
using  title  I  exemplary  projects  for  other  than  title  I  projects, 
we  believe  that  data  on  the  Network' s  impact  on  improving  title  I 
projects  should  distinguish  such  adoptions  from  those  that  replace 
or  modify  existing  title  I  projects. 

These  reporting  errors  resulted  in  the  accumulation  of  un¬ 
reliable  information  which  limited  its  usefulness. 

Network  actions 


To  improve  developer- demonstrators'  followup  on  project  adop¬ 
tions,  the  Network  in  April  1980  revised  its  operating  regulations 
to  specify  that  grant  applications  for  developer-demonstrators 
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will  be  examined  to  assess  the  extent  that  plans  "show  promise  of 
effective  postadoption  monitoring  and  evaluation  of  program  imple¬ 
mentation  and  resulting  benefits  at  the  adoption  sites." 

The  Network  regulations,  however,  did  not  specify  the  nature 
of  the  monitoring  and  evaluation  to  be  performed  or  the  data  to  be 
collected  during  the  postadoption  visits.  Network  officials  also 
pointed  out  that  the  current  funding  level  of  each  developer- 
demonstrator  is  not  sufficient  to  support  all  activities  for  which 
it  could  be  responsible. 

The  revised  Network  regulations  also  provide  that  the  Joint 
Dissemination  Review  Panel' s  approval  of  exemplary  projects  expires 
at  the  end  of  4  years.  Projects  applying  for  the  Panel' s  reap¬ 
proval  now  must  provide  evidence  that  adoptions  have  attained  the 
outcomes  originally  stated  by  the  project  developer.  These  require¬ 
ments  should  make  exemplary  project  data  more  current  and  provide 
data  on  the  results  of  some  project  adoptions.  The  Network  instruc¬ 
tions,  however,  do  not  specify  the  nature  or  type  of  data  to  be 
submitted. 

Revised  instructions  for  fiscal  year  1981  applications  for 
Network  developer-demonstrators  specify  that  applicants  seeking 
renewal  of  their  grants  must  provide  data  on  the  results  of  their 
operations,  including 

— the  number  of  students  in  direct  contact  with  teachers 
that  received  services  from  the  Network  developer- 
demonstrators  , 

— the  number  of  schools  provided  services  by  the  developer- 
demonstrators,  and 

— an  indication  as  to  whether  the  adoptions  have  produced 
a  significant  impact. 

These  data  will  give  the  Network  some  measures  of  the 
developer-demonstrators'  performance  during  the  operating  year. 
However,  the  reporting  format  does  not  indicate  whether  the  adop¬ 
tions  are  for  title  I  projects.  Consequently,  the  Network’s 
impact  on  title  I  projects  will  remain  unknown. 

Whether  the  Network' s  revised  regulations  and  instructions 
improve  developer- demonstrator  followup  activity  and  produce  data 
on  the  number  and  effectiveness  of  adoptions  will  depend  on  the 
extent  to  which  the  instructions  are  carried  out.  The  Network's 
instructions  do  not  detail  how  the  followup  and  reporting  should 
be  accomplished.  Also,  it  is  not  clear  how  the  developers  are  to 
report  the  status  of  adoptions  implemented  in  prior  years. 
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CHAPTER  4 


CONCLUSIONS,  RECOMMENDATIONS, 
AND  AGENCY  RESPONSE 


CONCLUSIONS 


Although  many  title  I  projects  are  improving  the  academic 
achievements  of  educationally  disadvantaged  students,  some  proj¬ 
ects  are  not  successful,  and  services  for  most  students  are  ter¬ 
minated  while  they  are  still  far  below  the  academic  achievement  of 
their  non-title  I  classmates.  Some  school  districts  have  adopted 
exemplary  projects  for  title  I  projects,  but  the  number  is  too 
small  to  have  much  impact  on  the  effectiveness  of  title  I  in  meet¬ 
ing  its  goals  nationwide. 

Inadequacies  in  title  I  evaluations  at  the  Federal,  State, 
and  local  levels  are  contributing  causes  for  the  relatively  few 
adoptions  for  title  I  projects.  ED's  title  I  evaluation  and  re¬ 
porting  system  provides  the  methodology  for  evaluating  the  title  I 
projects.  However,  ED  has  neither  prescribed  adequate  criteria 
for  State  and  local  officials  to  use  in  determining  the  adequacy 
of  their  projects  nor  effectively  monitored  the  States’  title  I 
administration.  The  State  education  agencies,  in  turn,  have  pro¬ 
vided  only  limited  monitoring  of  local  project  results.  Local 
title  I  officials  have  seldom  used  project  evaluations  as  a  tool 
for  project  planning  and  improvement.  Sometimes  inadequate  local 
evaluation  methodology  has  limited  the  reliability  and  usability 
of  evaluations  for  identifying  project  weaknesses.  Under  such 
circumstances,  relatively  few  local  school  districts  have  sought 
to  adopt  exemplary  title  I  projects. 

ED's  actions  to  implement  the  new  evaluation  models  should 
help  to  improve  the  reliability  and  comparability  of  title  I 
evaluations.  However,  project  improvement  depends  on  local  school 
officials'  receptivity  to  change.  Given  local  school  district 
officials'  satisfaction  with  their  title  I  projects  (see  p.  23) 
and  their  tendency  to  seek  information  that  supports  their  posi¬ 
tive  views  and  ignore  information  that  does  not,  few  school  dis¬ 
tricts  are  likely  to  adopt  exemplary  projects.  Consequently, 
unless  the  States  strongly  emphasize  using  title  I  evaluations  for 
project  improvements,  the  Network  is  unlikely  to  have  much  impact 
on  improving  title  I  projects. 

A  paucity  of  information  about  the  capabilities  of  exemplary 
projects  and  the  results  achieved  by  schools  adopting  such  proj¬ 
ects  also  has  led  local  school  officials  to  believe  these  projects 
are  no  better  than  their  own.  School  officials  would  be  more 
likely  to  adopt  exemplary  projects  if  more  comprehensive  informa¬ 
tion  were  provided  about  the  merits  of  such  projects  and  the  re¬ 
sults  achieved  by  schools  that  adopt  the  projects. 
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Because  implementing  educational  practices  is  a  complex, 
difficult  process,  we  believe  systematic  followup  of  adoptions 
is  needed  to  determine  whether  adopting  school  districts  have 
successfully  implemented  the  projects.  In  the  past,  a  serious 
shortcoming  has  been  the  Network's  lack  of  a  systematic  followup 
program  to  ensure  that  (1)  projects  are  properly  installed  and 
operating  in  the  manner  the  developer  believes  is  necessary  for 
success  and  (2)  additional  assistance  is  provided  when  the  projects 
are  faltering.  Revised  Network  instructions  emphasize  the  need 
for  monitoring  and  evaluation  at  the  adopting  sites  and  providing 
data  on  project  outcomes.  The  instructions,  however,  do  not 
specify  the  nature  of  the  evaluation  and  monitoring  to  be  performed 
or  the  data  to  be  provided.  Accordingly,  the  instructions'  value 
will  depend  on  how  aggressively  the  Network  pursues  compliance. 

A  more  extensive  followup  program  at  schools  adopting  Network 
projects  would  improve  the  accuracy  of  the  data  on  the  number  of 
adoptions.  Improved  data  would  help  the  Network  assess  the  inter¬ 
est  in  its  projects  and  the  need  for  improvements  in  its  operations. 

RECOMMENDATIONS 

To  improve  the  procedures  for  identifying  and  correcting 
title  I  project  weaknesses,  the  Secretary  of  Education  should 

— provide  substantially  increased  guidance  and  technical 
assistance  to  State  and  local  school  officials  in  develop¬ 
ing  criteria  for  assessing  the  effectiveness  of  title  I 
projects; 

— expand  ED’s  monitoring  efforts  for  assessing  whether 
State  education  agencies  are  (1)  meeting  their  responsi¬ 
bilities  to  evaluate  the  effectiveness  of  local  title  I 
projects  and  (2)  helping  local  school  districts  to  improve 
their  title  I  projects; 

— direct  State  education  agencies  to  (1)  assess  the  validity 
of  procedures  used  by  local  school  districts  to  evaluate 
their  title  I  projects,  (2)  prescribe  corrective  measures 
where  evaluation  procedures  are  inadequate,  and  (3)  help 
local  school  officials  use  evaluations  for  detecting  and 
correcting  project  weaknesses;  and 

— advise  State  education  agencies  to  encourage  local  school 
officials  to  use  Network  assistance  in  improving  ineffective 
and  inefficient  title  I  projects. 

To  convince  potential  adopters  of  the  merits  of  exemplary 
projects,  the  Secretary  of  Education  should 
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— direct  the  Network  to  provide  complete,  comparable,  and 
current  information  about  the  capabilities  of  the  Network's 
exemplary  projects,  including  (1)  their  impact  on  scholas¬ 
tic  achievement,  student  behavior,  and  sustained  growth, 

(2)  their  costs,  and  (3)  the  results  of  past  adoptions  by 
other  schools . 

To  provide  more  accurate  data  on  the  volume  of  title  I  adop¬ 
tions  of  exemplary  projects  and  ensure  that  adoptions  are  success¬ 
ful,  the  Secretary  of  Education  should  make  sure  that  the  Network's 
developer -demonstrators 

— provide  accurate  data  on  the  number  of  adoptions  and  the 
schools  affected, 

— determine  whether  the  projects  have  been  installed  cor¬ 
rectly  and  are  functioning  properly,  and 

— provide  for  additional  implementation  assistance  where 
needed . 

AGENCY  RESPONSE  AND  OUR  EVALUATION 

In  a  draft  of  this  report,  we  proposed  that  ED  prescribe 
specific  criteria  for  State  and  local  officials  to  use  in  assess¬ 
ing  the  effectiveness  of  title  I  projects.  In  its  April  21, 

1981,  response  to  our  draft  (see  app.  Ill),  ED  disagreed  with 
our  proposal. 

ED  said  there  is  no  legal  requirement  to  provide  criteria 
for  determining  what  levels  of  performance  are  satisfactory. 
Section  183(f)  provides  that  ED  title  I  evaluation  models  must 
specify  objective  criteria  for  use  in  evaluating  title  I  programs 
and  outline  techniques  for  producing  data  which  are  comparable  on 
a  statewide  and  nationwide  basis.  ED  contends  that,  under  this 
provision,  it  is  required  to  provide  criteria  or  standards  to 
help  school  districts  choose  methods  by  which  they  can  evaluate 
their  title  I  projects.  ED  believes  that  its  October  12,  1979, 
title  I  evaluation  regulations  provide  models  and  technical  stand¬ 
ards  for  evaluating  title  I  projects  and  that  this  satisfies  the 
statutory  requirement  to  provide  evaluation  criteria. 

However,  we  believe  that  ED's  obligation  in  fulfilling  its 
requirements  does  not  end  with  the  publication  of  evaluation 
models  and  technical  standards.  ED  should  place  increased  em¬ 
phasis  on  providing  technical  assistance  at  the  State  and  local 
levels  in  developing  criteria  for  assessing  the  effectiveness  of 
title  I  projects.  This  would  be  consistent  with  ED's  longstand¬ 
ing  policy  of  nonintervention  in  the  programmatic  decisionmaking 
process  at  the  State  and  local  levels. 
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According  to  ED,  implementation  of  the  title  I  evaluation 
system  will  permit  local  school  districts  to  assess  their  title  I 
achievement  gains  on  the  same  metric  or  scale  and  therefore  permit 
States  to  review  results  across  school  districts.  ED  said  that 
national  aggregation  will  enable  State  and  local  education  agen¬ 
cies  to  view  the  levels  of  their  gains  in  light  of  the  ranges  of 
gains  reported  nationally.  Additionally,  we  believe  that  ED 
should  use  the  national,  regional,  and  district  level  informa¬ 
tion  that  it  collects  to  provide  increased  guidance  to  State  and 
local  officials  concerning  criteria  for  assessing  the  effective¬ 
ness  of  title  I  projects,  and  we  have  revised  our  recommendation 
accordingly. 

ED  agreed  with  the  other  recommendations  in  this  report. 

Its  comments  are  discussed  below. 

Expand  monitoring  efforts  for 
assessing  State  evaluations  and 
assistance  to  school  districts 


ED  agreed  with  our  recommendations  and  said  that  it  places  a 
high  value  on  the  States'  responsibility  to  evaluate  the  effective¬ 
ness  of  local  title  I  projects  and  to  provide  assistance  to  local 
school  districts  to  improve  their  title  I  projects. 

ED  pointed  out  that  during  fiscal  year  1980  its  reviews  of 
State  education  agency  activities  devoted  greater  attention  than 
in  past  years  to  title  I  evaluation  and  that  it  intends  to  con¬ 
tinue  this  effort  to  ensure  that  States  and  local  education  agen¬ 
cies  meet  title  I  evaluation  requirements  as  well  as  to  improve 
the  quality  and  use  of  evaluations  at  State  and  local  levels. 

According  to  ED,  it  reviews  annually  the  States'  administra¬ 
tion  of  title  I,  assessing  how  well  they  are  meeting  their  evalua¬ 
tion  responsibilities  and  recommending  actions  to  improve  their 
overall  administration  of  title  I. 

Direct  State  education  agencies 
to  improve  the  validity  of 
evaluations  and  assure  their  use 
in  improving  title  I  projects 

ED  concurred  and  suggested  that  the  thrust  of  our  recom¬ 
mendation  was  being  met  through 

— ED  reviews  of  State  administration  of  title  I,  which 
include  recommendations,  as  needed,  for  improving 
evaluations; 
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— implementation  and  use  of  ED's  prescribed  evaluation 
models  and  standards;  and 

— the  evaluation  services  available  to  State  and  local  edu¬ 
cation  agencies  through  the  regional  Technical  Assistance 
Centers. 

These  efforts  should  help  improve  the  validity  of  local  eval¬ 
uations  and  make  them  more  usable  in  identifying  and  correcting 
project  weaknesses.  However: 

— Reports  on  ED  reviews  in  the  States  we  visited  seldom  indi¬ 
cated  that  evaluation  procedures  had  been  reviewed,  and 
none  of  the  reports  we  reviewed  made  recommendations  for 
correcting  inadequate  or  improper  evaluation  procedures. 

— Evaluation  procedures  were  technically  unsound  in  many  dis¬ 
tricts  we  visited  (including  districts  which  had  profes¬ 
sional  evaluators  and  had  implemented  the  required  evalua¬ 
tion  models). 

— The  Technical  Assistance  Centers  do  not  have  the  authority 
or  the  capability  to  monitor  and  correct  local  evaluation 
efforts.  They  can  advise  State  and  local  education  agencies 
only  when  help  is  requested.  They  cannot  conduct  the  evalu¬ 
ations,  perform  the  analyses,  or  interpret  the  evaluation 
results  for  the  States.  The  Centers  operate  in  a  support¬ 
ing  role,  not  one  of  supplanting  State  and  district  respon¬ 
sibilities  for  completing  their  annual  evaluations. 

— Technical  Assistance  Center  reports  suggest  that  the  Centers 
do  not  have  the  capacity  to  provide  all  the  assistance  needed 
by  the  local  school  districts.  For  example,  one  Center  re¬ 
ported  that  it  had  provided  assistance  to  less  than  32  per¬ 
cent  of  the  districts  in  its  area  as  of  September  30,  1979. 
Another  Center  reported  that  States  should  be  encouraged  to 
have  a  backup  system  for  training  local  school  district 
officials  in  evaluation  to  supplement  the  Center's  workshops. 

— State  monitoring  of  local  title  I  projects  has  been  compli¬ 
ance  oriented  and  has  been  ineffective  in  assuring  the 
validity  of  local  evaluations.  This  problem  was  strongly 
suggested  by  the  widespread  use  of  inappropriate  and  faulty 
evaluation  procedures  in  the  districts  we  visited. 

Accordingly,  we  believe  that  ED  needs  to  strengthen  its 
efforts  to  ensure  that  State  education  agencies  are  properly 
carrying  out  their  evaluation  responsibilities. 
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Encourage  the  use  of  the 
Network  for  title  I  projects 

ED  agreed  with  our  recommendation  and  stated  that  exemplary 
title  I  programs  disseminated  by  the  Network  should  be  included 
in  each  State's  repertoire  of  strategies  for  improving  teaching 
and  learning  in  title  I  schools.  ED  said  that  chief  State  school 
officers.  State  title  I  coordinators,  and  State  facilitators 
should  encourage  title  I  districts  to  examine  exemplary  title  I 
programs  diffused  by  the  Network  for  possible  solutions  to  their 
school  improvement  problems.  According  to  ED,  this  encouragement 
should  give  school  district  representatives  opportunities  to  meet 
face-to-face  with  the  developers  of  many  exemplary  programs.  This 
process,  according  to  ED,  gives  district  representatives  oppor¬ 
tunities  to  examine  materials  and  ask  questions  about  evaluation 
results,  installation  costs,  program  philosophy,  and  training 
requirements . 

ED  plans  to  continue  its  efforts  to  encourage  State  education 
agencies  to  work  with  local  education  agencies  to  assess  the  effec¬ 
tiveness  of  local  title  I  programs,  improve  local  projects  through 
the  adoption  of  exemplary  projects,  encourage  local  education 
agencies  to  critique  their  own  results,  and  consider  the  adoption 
of  exemplary  projects. 

ED  believes  encouraging  school  officials  who  are  not  satis¬ 
fied  with  their  programs  to  attend  awareness  conferences  will  in¬ 
crease  the  rate  of  title  I  adoptions.  ED  will  continue  to  work 
with  State  education  agencies  to  stimulate  interest  on  the  part  of 
local  education  agencies  to  analyze  results  of  their  current  proj¬ 
ects  and  to  consider  adopting  exemplary  projects,  if  appropriate. 

Provide  complete,  comparable,  and 
current  information  on  Network  projects 

ED  agreed  with  our  recommendation  and  stated  that,  as  required 
by  Network  regulations,  by  1984  for  each  project  to  retain  its 
exemplary  status,  project  data  must  be  submitted  to  the  Joint  Dis¬ 
semination  Review  Panel  documenting  that  the  results  are  still  out¬ 
standing,  that  there  have  been  adoptions  of  the  project,  and  that 
the  results  achieved  at  the  adopting  sites  are  also  outstanding. 

ED  said  that  the  use  of  title  I  evaluation  models  by  all 
local  education  agencies  will  result  in  Panel  submissions  with 
uniform  comparable  evidence  of  achievement  impact.  ED  added  that 
the  Panel  has  established  a  format  for  submission  of  both  new 
projects  and  those  being  revalidated  which  contains  a  specific 
format  for  reporting  data  on  project  startup  and  operating  costs. 
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ED  pointed  out  that,  although  sustaining  achievement  gains 
over  time  is  one  aspect  to  be  considered  in  reviewing  the  impact 
of  an  exemplary  project,  the  methodological  problems  in  conduct¬ 
ing  a  longitudinal  study  are  considerable.  However,  ED  believes 
that  some  evidence  of  sustained  effects  will  be  available  from 
each  exemplary  project  as  part  of  its  resubmission  to  the  Panel. 

ED  claimed  that  potential  adopters  raise  few  questions  about 
project  evaluation  results  and  doubted  whether  the  publication  of 
"more  elaborate  and  costly  information"  about  programs  will  result 
in  major  increases  in  the  number  of  adoptions.  However,  ED  con¬ 
curred  that  every  exemplary  project  should  make  information  avail¬ 
able  concerning  project  results  and  agreed  that  the  one-page 
descriptions  in  the  "Educational  Programs  That  Work"  catalog  can 
be  improved  to  provide  more  information. 

Strengthen  followup  of 
exemplary  project  adoptions 

ED  agreed  with  our  recommendations.  ED  said  it  plans  to  make 
greater  use  of  the  developer-demonstrators  to  monitor  claimed 
adoptions  in  each  State,  to  gather  information  on  adoptions 
(including  achievement  results),  and  to  maintain  contact  with 
adopting  sites.  ED  added,  however,  that  the  current  funding  level 
of  each  developer-demonstrator  is  not  sufficient  to  support  all  of 
the  activities  for  which  a  developer-demonstrator  could  be  respon¬ 
sible  in  all  50  States.  The  Network  is  attempting  to  provide  this 
support  within  its  funding  limitations.  ED  said  it  is  placing 
much  more  emphasis  on  the  quality  of  the  implementation  and  the 
persistence  of  installations  than  was  done  in  the  Network's  early 
years.  Also,  the  number  of  developer-demonstrators  funded  by  the 
Network  is  being  reduced  so  that  the  average  level  of  funding  can 
be  increased  slightly  to  make  additional  resources  available  for 
monitoring  and  followup  implementation  assistance  where  needed. 

ED  plans  to  require  more  information  in  the  future  from  each 
developer-demonstrator  concerning  the  status  of  adoptions  in 
order  to  strengthen  and  maintain  the  linkage  between  developer- 
demonstrators  and  school  districts  that  adopt  projects. 

According  to  ED,  it  recognizes  the  value  of  involving  the 
developer-demonstrator  as  an  active  participant  in  the  adoption 
process  and  followup,  and  the  developer-demonstrators  currently 
maintain  contact  with  their  adopting  sites  by  telephone  and  site 
visits.  ED  stated  that,  when  resubmitting  for  Joint  Dissemination 
Review  Panel  approval  before  1984,  each  developer-demonstrator  will 
have  to  include  achievement  data  from  five  adopting  sites  reflect¬ 
ing  the  diversity  of  sites  which  have  adopted  the  project. 

ED  stated  that  the  Network  collects  accurate  data  on  the 
number  of  adoptions,  the  number  of  schools,  and  the  number  of 
teachers  and  students  participating  in  the  adoptions  during  the 
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reporting  period  but  does  not  collect  data  on  which  adoptions 
are  title  I  projects  or  the  number  of  title  I  schools  affected. 

ED  explained  that,  when  the  present  adoption  reporting  forms  were 
cleared,  pressures  to  reduce  paperwork  and  reporting  requirements 
prevented  the  Network  from  collecting  information  on  the  numerous 
sources  of  funds,  including  title  I  funds,  used  to  install  Network 
projects.  ED  intends  to  request  clearance  from  the  Office  of  Man¬ 
agement  and  Budget  to  collect  information  on  the  particular  fund¬ 
ing  source  used  to  adopt  an  exemplary  project. 
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SELECTION  OF  SCHOOL  DISTRICTS 
FOR  SITE  VISITS 

We  selected  a  judgmental  sample  of  47  local  school  districts 
in  five  States  for  site  visits.  These  districts  were  visited  to 
ascertain  (1)  whether  local  school  districts  have  suitable  data 
for  assessing  the  need  for  title  I  program  improvement,  (2)  what 
action  districts  take  to  adopt  better  title  I  projects,  (3)  what 
impact  the  Network  has  on  improving  the  districts'  programs,  and 
(4)  what  improvements  district  officials  believe  are  needed  in 
disseminating  information  on  exemplary  projects. 

The  districts  selected  included  a  wide  variety,  ranging  from 
very  small  to  very  large  and  from  very  rural  to  highly  urbanized. 
Care  was  taken  to  ensure  that  the  number  of  school  districts  re¬ 
porting  above  average  districtwide  achievement  scores  approximated 
the  number  of  districts  reporting  below  average  scores.  Because 
title  I  was  created  to  assist  school  districts  affected  by  high 
concentrations  of  poverty,  we  did  not  select  the  more  affluent 
school  districts  for  site  visits.  For  this  determination,  school 
districts  having  smaller  than  average  proportions  of  low-income 
families  for  their  respective  States  were  not  selected. 

The  sample  of  47  school  districts,  which  had  a  combined  en¬ 
rollment  of  about  1,400,000  students,  included 

— 6  large  urban  districts  having  enrollments  over  50,000, 

— 17  medium-sized  districts  having  enrollments  between 
8,000  and  50,000,  and 

— 24  small  districts  having  enrollments  between  146  and 

8,000. 


The  seven  developer-demonstrators  we  visited  were  selected  in 
consultation  with  Network  officials  to  provide  a  range  of  experi¬ 
ences  in  terms  of  effectiveness  and  age  of  programs. 
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ANALYSIS  OF  THE  READING  ACHIEVEMENT 
GAINS  OF  TITLE  I  STUDENTS 


PURPOSE  OF  ANALYSIS 

We  analyzed  title  I  achievement  test  scores  in  14  school 
districts  to  (1)  examine  the  effectiveness  of  the  title  I  projects 
in  meeting  the  needs  of  educationally  disadvantaged  students, 

(2)  assess  the  needs  for  project  improvement,  and  (3)  compare 
local  project  results  with  reported  exemplary  project  results  to 
ascertain  whether  the  potential  benefits  of  adopting  exemplary 
projects  would  be  significant, 

SAMPLE  SELECTION 


From  the  47  school  districts  chosen  for  site  visits  (see 
app.  I),  we  selected  13  districts  for  detailed  analysis  of  title  I 
project  1/  achievement  test  results.  To  assure  balanced  represen¬ 
tation,  we  considered  school  district  size,  geographical  location, 
and  degree  of  urbanization.  We  also  considered  the  availability 
and  quality  of  achievement  test  data.  To  improve  the  geographical 
balance,  the  sample  of  13  districts  was  augmented  by  adding  a  dis¬ 
trict  not  included  in  the  original  47.  Our  analysis  of  the  14  dis¬ 
tricts  included  about  16,500  students  receiving  title  I  services 
in  approximately  340  schools. 

The  title  I  schools  we  analyzed  were  dispersed  among  the 
Nation's  four  geographic  regions,  2/  as  follows: 

— Northeast:  96  schools,  5,887  students  tested. 

— Southeast:  14  schools,  349  students  tested. 

— Central:  46  schools,  1,494  students  tested. 

— Western:  184  schools,  8,793  students  tested. 

Seven  of  the  school  districts  were  in  large  cities  having  over 
100,000  population,  four  were  in  smaller  cities  having  between 
10,000  and  100,000  population,  and  three  were  in  rural  areas. 

Four  of  the  districts  had  enrollments  of  over  50,000  students, 
seven  had  enrollments  of  8,000  to  50,000  students,  and  three  had 
enrollments  of  under  8,000  students.  Eight  districts  were  in 


1/ln  some  school  districts,  title  I  project  funds  are  supple¬ 
mented  with  State  and  local  funds. 

2/We  used  the  same  regions  defined  in  the  National  Assessment 
of  Educational  Progress. 
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counties  having  a  low-income  population  of  less  than  17  percent, 
four  were  in  counties  having  low-income  populations  between 
16.7  and  22.3  percent,  and  two  were  in  counties  where  more  than 
56  percent  of  the  population  was  low  income. 

ANALYSIS  METHODOLOGY 


Using  existing  achievement  test  scores  for  individual  students, 
we  tracked  the  progress  of  22  student  sample  groups  in  reading 
achievement  for  3  to  4  years,  depending  upon  the  availability  of 
records  in  each  district.  All  22  student  samples  were  selected 
from  grades  emphasized  in  the  districts*  title  I  programs.  In 
some  cases,  we  traced  the  students'  progress  over  a  period  of 
years  into  grade  levels  that  did  not  provide  title  I  services. 

All  samples  consisted  of  100  percent  of  the  students  receiving 
title  I  reading  assistance  in  the  grade  levels  selected  for  ini¬ 
tial  analysis.  Students  joining  tnese  groups  in  higher  grades 
during  subsequent  years  of  title  I  assistance  were  added  to  the 
sample  from  their  time  of  entry  into  the  project.  For  example, 
if  the  base  sample  consisted  of  all  third  grade  title  I  reading 
students  in  school  year  1975-76,  a  fourth  grade  student  enrolling 
in  title  I  reading  in  school  year  1976-77  would  be  included  in  the 
sample  from  the  time  of  enrollment.  These  additions  were  necessary 
to  show  project  results  based  on  all  students  served,  rather  than 
on  repeat  students  only. 

To  determine  the  need  for  improvement  in  local  title  I  proj¬ 
ects,  the  following  gains  were  computed  for  individual  groups  of 
students  and  their  respective  projects: 

— In-program  gains  achieved  by  students  during  a  school  year 
while  receiving  title  I  assistance. 

— Continuing  in-program  gains  achieved  by  students  over  con¬ 
secutive  years  of  title  I  service. 

— Sustained  gain  rates  maintained  by  students  after  their 
title  I  assistance  had  been  discontinued. 

The  achievement  gain  rates  were  analyzed  to  determine 
(1)  whether  the  students  were  gaining  or  losing  ground  on  their 
normal  achieving  peers  in  each  school  year,  (2)  the  cumulative 
effect  of  consecutive  years  of  title  I  assistance  on  student 
rates  of  achievement  gain,  and  (3)  the  effect  of  discontinuing 
title  I  assistance  on  student  gain  rates. 

Achievement  gain  measurement 

Student  achievement  gains  were  measured  by  subtracting  the 
pretest  from  the  posttest  achievement  level  for  the  period 
analyzed.  Expanded  standard  (scale)  scores  were  used.  Average 
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gains  for  the  projects  were  determined  by  aggregating  and  averag¬ 
ing  the  scores  of  students  served,  again  using  standard  (scale) 
scores.  Expected  or  normal  gain  was  defined  as  the  amount  of 
gain  that  the  average  achieving  student  at  the  same  age  and  grade 
level  would  achieve  for  the  period  of  instruction. 

Where  school  districts  used  tests  prepared  by  different  pub¬ 
lishers  within  the  several  testing  periods,  we  converted  all  test 
scores  to  expanded  standard  scores  of  the  test  battery  used  most 
frequently.  This  was  accomplished  using  an  intermediate  metric  or 
scale,  common  to  the  test  scores  being  converted  and  the  selected 
battery  scores.  The  broadest  category  of  reading  scores  (total 
reading)  rather  than  subtest  scores  (e.g.,  vocabulary  and  reading 
comprehension)  were  used. 

Limitations 


The  selected  sample  of  14  school  districts  is  not  large  enough 
to  enable  us  to  project  with  a  high  degree  of  confidence  that  the 
results  typify  title  I  projects  nationwide.  However,  our  sample 
was  carefully  selected  to  assure  a  broad  range  of  school  district 
sizes  and  environments,  and  we  believe  the  results  of  our  analysis 
should  be  indicative  of  many  of  the  Nation's  school  districts. 

In  the  districts  we  analyzed,  we  reviewed  the  test  data, 
checked  score  conversions,  and  considered  the  adequacy  of  such 
factors  as  norming  periods  of  the  tests.  Because  we  used  test 
data  available  in  the  school  districts  and  could  not  test  select¬ 
ing,  administering,  and  scoring,  we  cannot  guarantee  the  total 
accuracy  of  the  data. 

The  derived  test  results  may  reflect  some  achievement  gain 
inflation  due  to  gain  score  analysis  and  a  statistical  phenomenon 
known  as  "regression  toward  the  mean, "  wherein  students  scoring 
well  below  the  pretest  mean  tend  to  make  artificially  high  gains 
from  pretest  to  posttest  when  the  same  test  score  is  used  for 
student  selection  and  pretesting.  We  believe,  however,  that  the 
possible  gain  inflation  due  to  this  phenomenon  was  reasonably 
limited  in  our  analysis  because  we  followed  the  progress  of  stu¬ 
dents  over  several  years  and  the  same  test  score  was  generally 
not  used  for  initial  student  selection  and  pretest  measurement. 

In  addition,  in  our  opinion,  testing  was  usually  done  within  a 
reasonably  close  time  to  the  test  publisher's  norming  period. 

ED  expressed  concern  about  our  study's  reliance  on  the  grade 
equivalent  metric  or  scale  since  it  may  be  misinterpreted  and  can 
distort  the  measurement  of  achievement  if  misused.  According  to 
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a  report  prepared  for  the  National  Institute  of  Education  1/  on 
controversies  in  evaluating  compensatory  education,  a  basic  problem 
in  selecting  a  metric  or  scale  is  " *  *  *  one  of  validity  versus 
communicability;  the  more  technically  correct  units  are  not  neces¬ 
sarily  those  that  are  easiest  to  understand  or  directly  relevant 
to  decisionmaking."  The  report  claims  that  all  available  metrics 
or  scales  have  weaknesses  in  validity,  communicability,  or  both. 
However,  it  states  that  none  of  the  other  metrics  or  scales  has 
the  same  clarity  and  simplicity  of  meaning  as  grade  equivalents. 
Accordingly,  we  reported  achievement  results  in  grade  equivalents 
but  used  the  more  technically  correct  expanded  scale  score  to 
measure  academic  achievement. 


^/"Controversies  in  the  Evaluation  of  Compensatory  Education," 
American  Institutes  for  Research,  July  1977. 
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UNITED  STATES  DEPARTMENT  OF  EDUCATION 

WASHINGTON.  D  C.  »302 


ASSISTANT  SECRETARY 

FOR  ELEMENTARY  AND  SECONDARY  EDUCATION 

APR  2 1  1961 


Mr.  Gregory  J.  Ahart 
Director,  Human  Resources  Division 
United  States  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Ahart: 

The  Secretary  asked  that  I  respond  to  your  request  for  comments  on  your 
Draft  Report  entitled,  'Greater  Use  of  Exemplary  Education  Programs  Could 
Help  Improve  Education  for  Disadvantaged  Children." 

The  enclosed  comments  represent  the  tentative  position  of  the  Department  and 
are  subject  to  reevaluation  when  the  final  version  of  the  Report  is  received. 

While  we  concur  with  the  thrust  of  all  but  one  of  the  Report's  recommendations, 
we  wish  to  reiterate  our  concern  about  the  study’s  reliance  on  a  grade- 
equivalent  metric,  since  it  is  a  metric  often  misinterpreted.  In  fact,  as 
stated  on  page  40  of  the  Report,  GAO  is  aware  that  grade-equivalent  scores 
are  not  appropriate  for  determining  average  achievement  levels.  Therefore, 
it  is  unfortunate  that  this  metric  was  used. 


We  appreciate  the  opportunity  to  comment  on  this  Draft  Report  before  its 
publication. 


Sincerely, 


it  A?  f 

H.  Rodrigues  , J 

Acting  Assistant  Secretary 


Enclosure 


GAO  notes 


Page  references  in  this  appendix  may  not  correspond  to 
page  numbers  in  the  final  report. 
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Comments  of  the  Department  of  Education  on  the  Comptroller  General's  Proposed 
Draft  Report  to  the  Congress  Entitled,  "Greater  Use  of  Exemplary  Education 
Programs  Could  Help  Improve  Education  for  Disadvantaged  Children." 


CAP  Recommendation 


To  Improve  the  procedures  for  identifying  and  correcting  Title  I  project 
meeknesses,  the  Secretary  oi  Education  should  prescribe  specific  criteria  for 
State  and  local  school  officials  to  use  in  assessing  the  effectiveness  of 
title  1  projects. 

Department  Comment 

He  do  not  concur.  The  report  claims  that  the  Department  has  failed  to  provide 
States  with  adequate  criteria  for  assessing  Title  I  project  quality  and  effec¬ 
tiveness,  as  required  by  law.  This  position  is  based  upon  the  language  in 
Section  151(f)  of  the  Education  Amendments  of  1974,  renumbered  Section  183(f) 
of  the  Education  Amendments  of  1978,  20  U.S.C.  2833,  which  reads: 

"SPECIFICATION  OF  OBJECTIVE  CRITERIA. - 
The  models  developed  by  the  Commissioner 
shall  specify  objective  criteria  which 
shall  be  utilised  in  the  evaluation  of 
all  programs  and  shall  outline  techniques 
(such  as  longitudinal  studies  of  children 
Involved  in  such  programs)  and  methodology 
(such  as  the  use  of  tests  which  yield  com¬ 
parable  results)  for  producing  data  which 
are  comparable  on  a  statewide  and  nationwide 
basis."  (emphasis  supplied) 

Our  position  is  that  the  above-cited  provision  requires  the  Departsmnt  to 
provide  criteria  or  standards  to  assist  the  school  districts  in  choosing  methods 
by  tdilch  they  can  evaluate  their  Title  I  projects.  We  feel  that  the  final 
regulations  on  Title  I  evaluation  procedures,  issued  by  the  Department  on 
October  12,  1979,  fully  satisfy  this  statutory  requirement.  Those  regulations 
Include  provisions  on  technical  standards  to  be  used  by  local  educational  agencies 
(IX As)  in  evaluating  their  Title  I  projects,  and  provisions  on  the  use  of  evaluation 
models. 

GAO,  however,  reads  the  above-cited  provision  to  require  the  Department  to  pro¬ 
vide  criteria  for  assessing  the  quality  and  effectiveness  of  Title  I  projects. 

He  have  researched  the  legislative  history  of  the  Title  I  evaluation  provisions 
and  found,  contrary  to  GAO's  assertion,  no  indication  that  Congress  intended  the 
Department  to  provide  these  kinds  of  criteria. 

The  Report  goes  on  to  criticise  the  Department  for  falling  to  provide  criteria  for 
determining  what  levels  of  performance  are  satisfactory  and  what  levels  are  in¬ 
adequate.  Aside  from  the  fact  that  there  is  no  legal  requirement  for  providing 
these  criteria,  to  do  so  would  conflict  with  the  Department's  longstanding  policy 
on  non-intervention  in  the  programmatic  decisionmaking  process  at  the  State  and 
local  levels.  In  fact.  Section  432  of  the  General  Education  Provisions  Act 
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prohibits  the  Department  from  exercising  any  direction  or  control  over  the 
programming  and  administration  In  the  State  and  local  agencies.  The  Department 
recognizes  that  different  school  districts  have  different  needs  and  concerns, 
an)  therefore  feels  that  such  questions  as  "What  is  the  satisfactory  level  ci 
performance?"  and  "What  is  an  effective  Title  I  program?"  are  best  answered  by 
the  State  educational  agencies  (SEAs)  and  LEAs  most  aware  of  those  needs  and 
concerns. 

ED  has  concentrated  Its  activities  on  assisting  SEAs  and  LEAs  to  conduct  valid 
and  ultimately  usable  evaluations  of  Title  I  projects.  Our  strategy  has  been 
evolutionary.  Bearing  in  mind  that  the  majority  of  Title  I  districts  are  quite 
small  and  probably  do  not  have  trained  evaluators,  we  have  concentrated  our  re¬ 
sources  on  fundamentals — identifying  appropriate  tests;  administering,  scoring, 
and  analyzing  the  results  accurately;  correctly  implementing  an  evaluation 
model;  reviewing  an)  interpreting  the  results;  and,  finally,  making  judgments  about 
project  components.  After  Implementing  the  Title  I  Evaluation  and  Reporting  System 
(TIERS)  for  several  years,  many  SEAs  and  LEAs  are  beginning  to  monitor  closely  the 
patterns  of  results  from  their  Title  I  evaluations.  They  are:  (1)  developing 
monitoring  plans  to  review  the  particularly  successful  and  unsuccessful  projects, 

(2)  asking  questions  relating  to  the  implementation  of  projects  that  are  extreme, 
an)  (3)  beginning  to  identify  strengths  and  weaknesses  of  their  projects.  We  will 
provide  technical  assistance,  as  requested,  to  conduct  process  evaluations  of  un¬ 
usually  effective  or  ineffective  projects,  and  to  assist  in  identifying  and  dis¬ 
seminating  effective  local  practices  and  identifying  and  correcting  program  weak¬ 
nesses.  Implementation  of  the  TIERS  will  permit  LEAs  to  assess  Title  I  achievement 
gains  of  each  Title  I  school  and  SEAs  to  review  results  across  LEAs  since  the  re¬ 
porting  metric  will  be  uniform  and  the  results  comparable.  ED's  national  aggregation 
will  enable  SEAs  and  LEAs  to  view  the  levels  of  their  gains  in  light  of  the  ranges  of 
gains  reported  nationally. 

The  strategy  ED  has  adopted  does  produce  an  interpretable  measure  of  whether  Title  I 
projects  are  effective.  Interpretation  of  results,  however,  should  and  does  differ 
from  site  to  site.  The  Title  I  evaluation  models  yield  measures  of  achievement  gain, 
i.e.,  how  much  Title  I  students  gain  above  what  would  have  been  expected  in  the  ab¬ 
sence  of  Title  I.  A  positive  gain  from  an  evaluation  shows  that  a  project  has  suc¬ 
cessfully  increased  the  rate  of  growth  of  its  participants.  In  theory,  the  larger  the 
gain,  the  better  the  program.  A  negative  "gain"  indicates  that  the  Title  I  partici¬ 
pants  are  growing  at  a  slower  rate  than  their  non-Title  I,  but  similarly  disadvantaged, 
peers.  Specification  of  criteria  to  separate  the  satisfactory  projects  from  the  un¬ 
satisfactory  projects  is  technically  unsound,  educationally  indefensible,  and  an 
intrusion  into  the  legal  authority  of  the  States  and  their  local  school  districts. 

GAO  Recommendation 


To  improve  the  procedures  for  identifying  and  correcting  Title  I  project  weak¬ 
nesses,  the  Secretary  of  Education  should  expand  ED's  monitoring  efforts  for 
assessing  whether  State  education  agencies  are  (1)  meeting  their  responsibilities 
to  evaluate  the  effectiveness  of  local  Title  I  projects,  and  (2)  assisting  local 
school  districts  to  improve  their  Title  I  projects. 
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Department  Coament 

We  concur.  ED  places  a  high  value  on  the  State's  responsibility  to  evaluate  the 
effectiveness  of  local  Title  I  projects  and  to  provide  assistance  to  local  school 
districts  to  Improve  their  Title  X  projects. 

ED  teams  review  annually  each  State's  administration  of  Title  I,  assessing  how  well 
States  are  meeting  their  evaluation  responsibilities  and  recommending  actions  to 
be  taken  by  the  SEA  to  improve  its  overall  administration  of  Title  1. 

In  preparing  for  their  onsite  monitoring  visits  the  ED  review  teams  examine  State 
evaluation  reports  and  a  sample  of  local  agency  reports  for  compliance  with 
evaluation  requirements  and  to  assess  program  quality.  They  also  review  Technical 
Assistance  Center  (TAC)  reports  which  often  suggest  evaluation  improvements  to  be 
made  by  the  State.  These  are  discussed  with  program  and  evaluation  specialists 
in  the  SEA  for  the  purpose  of  outlining  team  onsite  strategies. 

While  visiting  State  agencies,  the  teams  review  evaluation  practices,  including 
technical  assistance  provided  and  the  use  made  of  local  evaluations  during  the 
application  preparation  and  approval  process.  At  local  agencies  they  review 
evaluation  practices  and  discuss  the  use  of  evaluation  results  as  a  basis  for 
program  Improvements. 

On  October  12,  1979,  the  final  Title  I  evaluation  regulations  were  published. 

ED  review  teams  were  thus  able  to  assess  State  and  local  adherence  to  the  standards 
and  models  contained  therein.  During  FY  1980  ED  reviews  devoted  greater  attention 
than  in  past  years  to  the  area  of  Title  I  evaluation  and  their  reports  to  the 
States  reflect  this  increased  attention.  ED  Intends  to  continue  this  effort  to 
Insure  that  SEAs  and  LEAs  meet  Title  I  evaluation  requirements  as  well  as  to 
Improve  the  quality  and  use  of  evaluations  at  State  and  local  levels*  Also,  for 
the  fifth  year  now,  the  Title  I  TACs  are  assisting  SEAs  and  their  LEAs  in  im¬ 
proving  the  quality  of  evaluations  of  Title  I  projects  anl  in  using  evaluation 
results  for  program  improvement.  This  assistance  takes  the  form  of  materials 
development  related  to  needs  assessment,  student  selection,  testing,  and 
product  and  process  evaluation.  In  response  to  requests  from  SEAs  and  LEAs, 
workshops  related  to  test  selection,  evaluation  design,  needs  assessment,  instruc¬ 
tional  decisionmaking,  and  evaluation  are  developed  and  presented. 

The  TAC  services  are  provided  free  of  charge  to  State  and  local  education  personnel 
to  assist  on  a  variety  of  Title  I  evaluation  Issues.  As  a  result  of  the  TAC  services 
and  the  development  of  the  reporting  system,  State  and  local  education  personnel 
are  conducting  more  complete  evaluations  of  their  programs.  These  evaluations  are 
of  a  continued  higher  quality  because  the  system  stresses  evaluation  data  quality 
control  and  the  TACs  provide  technical  assistance  to  support  this  effort.  The 
development  of  sustained  gains  evaluation  at  the  local  level  is  providing  local 
personnel  with  data  about  the  effect  of  Title  I  over  a  longer  period  of  time  than 
one  year.  SEAs  and  LEAs  are  devoting  increasing  attention  to  the  area  of  process 
evaluation.  This  phase  of  evaluation  provides  contextual  input  into  the  instruc¬ 
tional  decision  process.  With  this  increased  capacity,  State  and  local  education 
personnel  are  able  to  evaluate  programs  more  effectively  and  use  these  data  for 
program  development. 
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GAO  Recommendation 

To  Improve  the  procedures  for  Identifying  and  correcting  Title  I  project  weaknesses, 
the  Secretary  of  Education  should  direct  State  education  agencies  to  (1)  assess  the 
validity  of  procedures  used  by  local  school  districts  to  evaluate  their  Title  I 
projects,  (2)  prescribe  corrective  measures  where  evaluation  procedures  are 
inadequate,  and  (3)  assist  local  school  officials  In  the  use  of  evaluations  for 
detection  and  correction  of  project  weaknesses. 


Department  Comment 

We  concur.  As  indicated  in  our  response  to  the  monitoring  recommendation,  ED  teams 
review  State  administration  of  Title  I,  including  site  visits  to  a  number  of  LEAs 
each  year  to  observe  local  project  operation.  As  appropriate,  recommendations  are 
made  for  SEA  improvement  in  the  area  of  evaluation. 

In  compliance  with  Section  151  of  Public  Law  93-380,  the  Education  Amendments  of 
1974,  ED  developed  evaluation  models  and  standards  for  use  by  SEAs  and  LEAs.  On 
October  12,  1979,  final  regulations  were  published  specifying  models  for  evaluating 
the  effectiveness  of  LEA  projects  providing  instructional  services  in  reading, 
language  arts,  or  mathematics.  However,  as  early  as  1978,  although  not  yet  federally 
mandated,  approximately  5,500  school  districts  were  in  the  process  of  Implementing 
one  of  the  models.  The  experiences  of  ED,  the  SEAs,  and  the  l£As  in  using  these 
models  were  disseminated  in  a  pamphlet  entitled,  "The  U.S.  Office  of  Education  Models 
to  Evaluate  E.S.E.A.  Title  I:  Experiences  After  One  Year  of  Use." 

In  addition,  the  ten  regional  TACs  prepare  support  materials  to  use  in  conjunction 
with  their  workshops  and  personal  (onsite,  telephone,  written)  consultations. 
Materials  produced  include  topical  papers  (e.g. ,  out-of-level  testing);  simulation 
exercises  (e.g.,  test  selection;  reporting  forms  completion);  checklists  for 
implementing  a  particular  model;  handouts  (e.g.,  test  charts);  and  transparencies 
for  use  with  an  overhead  projector. 

The  TACs  keep  the  States  aware  of  the  availability  of  technical  assistance  through 
Regional  Coordinating  Council  meetings,  scheduled  meetings  with  State  contacts  on 
an  individual  or  group  basis,  and  distribution  of  regional  newsletters. 

Any  SEA  or  LEA  may  request  services  from  ED  or  directly  from  its  TAC.  Assistance 
is  often  provided  as  a  follow-up  to  ED  review  team  findings  and  recommendations. 

ED  will  continue  to  support  efforts  in  these  areas. 


GAO  Recommendation 


To  improve  the  procedures  for  Identifying  and  correcting  Title  I  project  weaknesses. 
the  Secretary  of  Education  should  advise  State  education  agencies  to  encourage  local 
school  officials  to  use  the  assistance  available  through  the  Network  for  improving 
Ineffective  and  inefficient  Title  I  projects. 


52 


APPENDIX  III 


APPENDIX  III 


Department  Comment 

We  concur.  Exemplary  Title  I  programs  disseminated  by  the  National  Diffusion 
Network  (NDN)  should  be  included  in  each  State's  repertoire  of  strategies  for 
Improving  teaching  and  learning  in  Title  1  schools.  Chief  State  School  Officers, 
State  Title  I  Coordinators,  and  State  Facilitators  should  encourage  Title  1 
districts  to  examine  exemplary  Title  1  programs  diffused  by  the  NDN  for  possible 
solutions  to  their  school  Improvement  problems.  Based  upon  previous  studies, 
this  encouragement  should  provide  school  district  representatives  with  opportuni¬ 
ties  to  meet  face-to-face  with  the  developers  of  many  exemplary  programs.  This 
process  gives  district  representatives  opportunities  to  examine  materials  and  ask 
questions  about  evaluation  results,  Installation  costs,  program  philosophy,  and 
training  requirements.  ED  plans  to  continue  its  efforts  to  encourage  SEAs  to  work 
with  LEAs  to  assess  the  effectiveness  of  local  Title  I  programs,  to  Improve 
local  projects,  as  needed,  through  the  adoption  of  exemplary  projects,  and  to 
encourage  LEAs  to  critique  their  own  results  and  to  consider  the  adoption  of  an 
exemplary  project,  as  appropriate.  Encouragement  and  guidance  is  provided  to  SEAs 
and  LEAs  through  ED’s  annual  national  dissemination  meetings,  ED  review  team 
monitoring,  and  materials  developed  and  distributed  nationally.  After  examining 
several  alternatives,  the  district  can  choose  the  program  that  best  fits  its  needs 
and  resources  and  arrange  for  staff  training.  Of  course,  not  all  districts  will 
find  programs  in  the  NDN  to  fit  their  needs  and  may  have  to  turn  to  other  sources. 

GAO  Recommendation 


To  convince  potential  adopters  of  the  merits  of  exemplary  projects,  the  Secretary 
ol  Education  should  direct  the  Network  to  provide  complete,  comparable,  and  current 
information  about  the  capabilities  of  the  Network's  exemplary  projects,  including 

(1)  their  Impact  on  scholastic  achievement,  student  behavior  and  sustained  growth, 

(2)  their  costs,  and  (3)  the  results  of  past  adoptions  by  other  schools. 

Department  Comment 

We  concur.  A  brief  summary  of  this  Information  is  included  in  the  program 
description  in  Educational  Programs  That  Work,  a  publication  Issued  annually  by  the 
NDN  program.  Also  Included  in  the  publication  is  the  name  and  telephone  number  of 
the  project  director.  A  telephone  call  or  letter  can  be  used  to  obtain  additional 
information  about  a  program.  In  addition,  all  of  this  information  can  be  obtained 
face-to-face  from  the  developer  of  an  exemplary  program  during  ED  or  SEA-sponsored 
awareness  conferences.  The  SRI  International  evaluation  of  NDN  found  that  most 
decisions  to  Install  an  NDN  exemplary  program  are  made  as  a  result  of  representa¬ 
tives  from  a  school  district  either  visiting  the  program  to  see  it  in  operation,  or 
attending  an  awareness  conference  and  talking  with  the  developer  of  the  program. 

We  will  continue  to  promote  national  and  State  level  awareness  conferences  so  that 
developers  and  interested  LEA  representatives  can  meet  face-to-face. 
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As  required  by  the  NDN  regulations,  by  1984  for  each  exemplary  project  to  retain 
its  exemplary  status,  project  data  must  be  submitted  to  the  Joint  Dissemination 
Review  Panel  (JDRP)  documenting  that  the  results  are  still  outstanding,  the*  there 
have  been  adoptions  of  the  project,  and  that  the  results  achieved  at  the  adopting 
sites  are  also  outstanding.  Use  of  the  Title  I  evaluation  models  by  all  LEAs  will 
result  in  JDRP  submissions  with  uniform,  comparable  evidence  of  achievement  Impact. 
The  format  for  submissions  to  the  JDRP,  both  for  new  projects  and  for  those  being 
revalidated,  is  established  by  the  JDRP  and  contains  a  specific  format  for  reporting 
data  on  project  start-up  and  operating  costs. 

The  Title  I  statute  (P.L.  95-561,  Section  124(g))  requires  each  LEA  to  examine 
during  each  three-year  period  the  sustained  effects  of  Title  I  services.  ED  has 
developed  and  the  TACs  are  disseminating  alternatives  for  LEAs  to  employ  to  meet 
this  requirement  and  to  use  the  results  for  project  improvement.  Although 
sustaining  achievement  gains  over  time  is  one  aspect  of  project  success  to  be 
considered  in  a  review  of  the  impact  of  an  exemplary  project,  the  methodological 
problems  in  conducting  a  longitudinal  study  are  by  no  means  trivial.  However, 
some  evidence  of  sustained  effects  will  be  available  from  each  exemplary  project 
as  part  of  its  resubmission  to  the  JDRP. 

Our  experience  with  the  NDN  indicates  that  few  decisions  to  install  a  program  were 
made  as  a  result  of  reading  printed  materials.  Developer-Demonstrator  project 
directors  report  that  very  few  questions  are  asked  by  potential  adopters  about 
evaluation  results  obtained.  We  concur  that  every  exemplary  project  should  make 
this  information  available  upon  request,  and  we  believe  that  the  one-page  descrip¬ 
tions  in  Educational  Programs  That  Work  can  be  Improved  to  provide  more  information. 
However,  if  school  officials  in  41  out  of  the  47  school  districts  visited  (87Z)  were 
satisfied  with  their  Title  I  programs  (page  36),  we  doubt  that  the  publication  of 
more  elaborate  and  costly  information  about  programs  will  result  in  major  increases 
in  the  number  of  adoptions.  Data  on  the  effectiveness  of  the  exemplary  programs 
were  available,  but  school  officials  apparently  chose  not  to  make  inquiries.  We  do 
believe  that  encouraging  school  officials  in  the  six  districts  out  of  forty-seven 
that  were  not  satisfied  with  their  programs  to  attend  awareness  conferences  will 
increase  the  current  rate  of  Title  I  adoptions.  ED  will  continue  to  work  with  SEAs 
to  stimulate  interest  on  the  part  of  lEAs  to  analyze  results  of  their  current  proj¬ 
ects  and  to  consider  the  merit  of  adopting  an  exemplary  project,  if  appropriate. 

GAO  Recommendation 


To  provide  more  accurate  data  on  the  volume  of  Title  1  adoptions  of  exemplary  proj¬ 
ects  and  insure  that  adoptions  of  the  projects  are  successful,  the  Secretary  of 
Education  should  make  sure  that  the  Network's  developer-demonstrators' followups  of 
project  adoptions  are  successful  in 

—  providing  accurate  data  on  the  number  of  Title  I  adoptions  and  the 
schools  affected, 

—  determining  whether  the  projects  have  been  installed  correctly  and  are 
functioning  properly,  and 

—  providing  additional  implementation  assistance  where  needed. 
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Depwftnt  Comment 

We  concur.  NDN  does  collect  eccurete  date  on  the  number  of  adoptions,  the  number 
of  schools,  and  the  nunber  of  teachers  and  students  participating  in  the  adoptions 
during  the  reporting  period.  It  does  not  collect  data  on  which  adoptions  are 
Title  1  projects  or  on  the  nunber  of  Title  1  schools  affected.  When  the  present 
reporting  fonts  were  cleared,  pressures  to  reduce  paperwork  and  reporting  require¬ 
ments  prevented  NDN  from  collecting  Information  on  the  nunerous  sources  of  funds, 
including  Title  I  funds,  used  to  Install  NDN  programs.  We  do  intend  to  request 
from  the  Office  of  Management  and  Budget  clearance  to  collect  Information  on  the 
particular  funding  source  used  to  adopt  an  exemplary  project. 

We  recognize  as  well  the  value  of  involving  the  Developer-Demonstrator  (DD)  as  an 
active  participant  in  the  adoption  process  and  follow-up.  Using  existing  resources, 
ED  plana  to  make  greater  use  of  the  DDs  to  monitor  claimed  adoptions  in  each  State, 
to  gather  information  on  adoptions  (including  achievement  results),  and  to  maintain 
contact  with  adopting  sites  over  time.  Presently,  DDs  maintain  contact  with  their 
adopting  sites  by  telephone  and,  if  within  the  DD's  State,  by  on-site  visits.  When 
resubmitting  for  JDRP  approval  prior  to  1984,  each  DD  must  include  achievement  data 
from  five  adopting  sites  reflecting  the  diversity  of  sites  which  have  adopted  the 
project.  The  current  funding  level  (approximately  $40,000)  of  each  DD  is  not 
sufficient  to  support  all  activities  in  50  States  for  which  Developer-Demonstrators 
could  be  responsible,  auch  as: 

°  participating  in  awareness  conferences  so  that  potential  adopters 
can  obtain  information  directly  from  the  developer; 

*  conducting  training  for  personnel  in  districts  that  have  made 
commitments  to  install  their  program; 

*  monitoring  the  program  after  it  has  been  installed;  and 

*  providing  follow-up  technical  assistance  where  needed. 

NDN  Is  attempting  to  provide  these  services  within  its  funding  limitations. 

Much  more  emphasis  is  being  placed  upon  the  quality  of  implementation  and  the 
persistence  of  installations  than  was  the  case  in  the  early  years  of  NDN.  The 
number  of  Developer-Demonstrator  Projects  funded  by  NDN  is  being  reduced  so  that 
the  average  level  of  funding  can  be  increased  slightly  to  make  additional  resources 
available  for  monitoring  and  follow-up  implementation  assistance  where  needed.  In 
the  future,  ED  will  require  more  information  from  each  DD  on  the  status  of  adoptions. 
Thus,  the  linkage  between  DD  and  adopters  will  be  strengthened  and  maintained. 
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